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ABSTRACT 


Performance  measures  from  the  first  major  experiment  in  a 
series  on  tactical  decision-making  for  threat  evaluation  and  action  selection 
in  aerospace  surveillance  are  described.  Two  groups  of  experimental 
commanders  performed  under  several  levels  of  target  track  load  and  threat 
complexity  conditions.  The  task  of  the  commanders  was:  (1)  to  minimize 
damage  to  the  weapon  areas,  (2)  to  destroy  a  maximum  number  of  threaten¬ 
ing  vehicles,  (3)  to  conserve  counter  weapons  so  as  to  consume  a  minimum 
of  forces  consistent  with  objectives  (1)  and  (2)  above  and  (4)  to  develop  his 
own  strategy  under  constraints  imposed  by  the  ground  rules. 

In  order  to  provide  some  standard  performance  criteria,  a  sample 
of  the  experimental  problems  was  solved  analytically  so  as  to  represent 
both  "good"  and  "poor"  automated  decision  making  and  "idealized"  or  maxi¬ 
mum  human  performance.  Human  empirical  scores  compared  favorably 
with  analy iical  performance  measures.  Human  empirical  performance 
continuously  improved  as  a  function  of  number  of  system  runs. 
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INTRODUCTION 


This  report  is  one  of  a  series  concerned  with  research  in  the 

evaluation  of  surveillance  situations  and  the  selection  of  appropriate 

actions.  This  particular  portion  of  the  work  involves  experimental 

l 

investigations  of  executive  control  function  in  simulated  environments 
generally  representative  of  future  tactical  aerospace  operational  sit¬ 
uations.  These  investigations  are  a  part  of  the  research  activities  in 
information  processing  conducted  at  L.G.  Hanscom  Field  by  the 
Detection  Physics  Laboratory,  AFCRL,  OAR  and  the  Operational 
Applications  Laboratory,  ESD,  Most  of  this  research  has  been 
carried  out  under  projects  documented  in  Technical  Area  760B,  Sur¬ 
veillance  Techniques,  of  ARDC's  Applied  Research  Program. 

Although  rather  fundamental  in  nature  (i.  e.  ,  in  the  earlier  or  more 
basic  phase 8  of  the  applied  research  cycle)  this  work  has  been  biased 
toward  the  types  of  activities  involved  in  USAF  surveillance  operations. 

The  purpose  of  this  particular  section  is  to  define  the  nature 
of  information  processing  as  an  integral  part  of  surveillance,  to 
establish  the  general  boundaries  or  limits  of  the  areas  considered,  and 
to  outline  some  of  the  underlying  assumptions  and  general  goals  of 
this  series  of  experimental  investigations. 

The  effectiveness  of  the  U.  S.  Air  Force  in  any  given  mission 
depends  partly  on  the  type  and  quantity  of  operational  tools  available 
at  that  time;  e.  g.  ,  the  weapons,  aircraft,  missiles,  and  supplies. 

*This  introduction,  principally  written  by  W.  H.  Vance,  AFCRL 
Project  Scientist  for  Project  4690,  is  essentially  the  justification  for 
the  work  on  tactical  decision  making  and  is  the  vehicle  which  dictated 
the  direction  of  the  OAL  effort  under  Project  4690. 


However,  it  also  depends  on  when  and  how  these  tools  are  used,  i.e.  , 
on  command  decisions.  To  a  very  large  degree,  the  effectiveness  of 
command  decisions  is  dependent  on  the  timely  availability  of  pertinent 
information  about  the  operational  situation,  and  skillful  evaluation  of 
this  situation  in  terms  of  the  mission.  The  need  to  know  the  location 
of  the  enemy,  his  disposition  and  strength  (as  well  as  similar  infor¬ 
mation  about  our  own  forces)  is  as  old  as  war  itself.  Collecting  and 
interpreting  such  data  in  an  aerospace  situation  is  the  function  of 
surveillance.  Surveillance  provides  the  basis  for  short  term,  immed¬ 
iate,  or  "tactical"  decisions  --  to  scramble  certain  interceptors,  or 
to  launch  missiles,  or  to  divert  bombers  to  a  new  target,  or  to  release 
SAC  for  a  full  scale  retaliatory  raid  --  i.  e.  ,  when  and  how  to  utilize 
existing  capability.  Thus,  surveillance  represents  an  important 
element  of  many  military  command  decisions;  in  fact  in  operational 
aerospace  situations  it  becomes  the  critical  element  of  the  command 
decision  process. 

Surveillance  is  not  a  new  function.  It  has  been  performed  in 
the  past;  the  Air  Force  has  a  certain  surveillance  capability  today;  it 
will  need  a  vastly  improved  surveillance  capability  in  the  future.  The 
operational  surveillance  capability  today  is  determined  by  the  equip¬ 
ment  and  personnel  now  available.  Providing  the  best  possible 
surveillance  capability  for  the  next  few  years  is  the  mission  of  the 
development  program,  i.e,,  the  SystemProgramOffice s.  Providing 
fundamental  knowledge,  concepts,  methods,  and  techniques  on  which 
to  base  development  of  equipment  systems  at  some  future  time  is  the 
purpose  of  our  research  program. 
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As  has  been  pointed  out,  surveillance  has  been  performed  in 
the  past,  and  is  performed  today.  Military  commanders  have  always 
faced  problems  in  optimum  utilization  of  their  forces  --  they  have 
always  had  difficulties  in  "acquiring  information,"  in  "evaluating 
the  situation,"  and  in'hiaking  command  decisions."  In  recent  years 
there  has  been  great  emphasis  on  the  design  and  construction  of 
equipment  and  systems  to  assist  military  commanders  in  these 
functions.  Although  great  progress  has  been  made  in  some  areas, 
there  is  no  doubt  that  our  present  capability  for  collecting,  process¬ 
ing,  and  evaluating  information,  and  effectively  controlling  military 
activities  is  entirely  inadequate.  Even  more  serious,  any  reasonable 
projection  of  future  problems  indicates  that  the  situation  is  likely 
to  get  worse. 

Why  is  this  so?  Why  are  we  concerned  with  these  same  old 
problems?  Since  we  managed  to  solve  them  well  enough  in  past 
operations,  what  is  so  different  about  today's  situations,  and 
tomorrow's?  Some  study  has  been  made  of  questions  of  this  type, 
but  a  detailed  discussion  is  beyond  the  scope  of  this  report.  For 
the  present,  a  brief  mention  of  a  few  of  the  more  important  factors 
affecting  surveillance  and  command  decision  problems  will  suffice: 

1.  The  availability  of  long  range  delivery  systems  means 
that  aerospace  operations  can  be  conducted  over  much  greater 
ranges  than  ever  before.  Consequently,  operational  situations  must 
be  observed,  evaluated,  and  controlled  over  larger  volumes  of 
space,  many  of  them  on  a  global  basis. 
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2.  The  high  speeds  of  delivery  systems  have  compressed  the 
total  time  available  for  the  entire  information-evaluation-decision- 
action  process  to  a  matter  of  minutes.  In  many  vital  operational  sit¬ 
uations  at  least,  we  can  no  longer  afford  the  luxury  of  staff  conferen¬ 
ces,  or  extensive  reflection  and  consideration  of  the  situation  before 

a  decision  is  made. 

3.  Many  modern  delivery  systems  are  unmanned  and  unre- 
callabie,  so  that  once  action  is  initiated  (e.  g.  ,  to  launch)  there  is 
little  or  no  opportunity  for  modification  of  the  decision.  Thus,  while 
there  is  need  for  faster  decisions,  they  also  must  be  reliable. 

4.  The  advent  of  thermonuclear  weapons,  with  vastly  greater 
destructive  capability  than  ever  existed  before,  has  significantly 
affected  the  evaluation-decision  process  in  several  ways,  for  example, 
the  terrible  consequences  of  a  "wrong"  decision.  The  problems  are 
also  complicated  by  the  unknowns  caused  by  the  lack  of  experience 

in  the  actual  use  of  such  weapons;  consider  that  only  two  atomic  bombs 
have  ever  been  exploded  in  anger,  both  of  them  quite  puny  by  today's 
standards ! 

5.  The  necessity  to  evaluate  and  control  situations  larger  in 
scope  (1  above)  with  a  shorter  reaction  time  (2  above)  has  p  ished  the 
critical  decision  and  control  level  higher  and  higher  in  the  organiza¬ 
tional  hierarchy.  Similarly,  the  unfamiliar  nature  of  nuclear  war  and 
the  potential  impact  of  "wrong"  decisions  (3  and  4  above)  have  strongly 
reinforced  the  traditional  desire  for  centralized,  high  level  decision¬ 
making  and  control.  Although  similar  problems  have  existed  before 
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at  "tactical  command"  levels,  their  emergence  at  the  highest  "national" 
levels  is  new. 


6,  The  same  weapons  which  emphasize  the  urgency  for  re¬ 
liable  centralized  control  create  a  host  of  impediments  to  achieving 
it.  Information  must  be  collected  over  greater  volumes  and  comm¬ 
unicated  over  long  distances,  thus  tending  to  reduce  the  quality, 
timeliness,  and  reliability  of  the  data;  the  effects  of  delays  and  errors 
in  decision-making  are  magnified;  sensing  devices,  communication 
links,  and  commanders  are  more  vulnerable,  etc. 

Thus,  even  from  this  brief  discussion,  it  is  apparent  that 
rapid  advances  in  military  technology  have  created  a  whole  host  of 
new  problems  in  the  collection,  evaluation,  and  processing  of  infor¬ 
mation  for  control  of  aerospace  operations.  In  the  most  stringent 
cases,  information  must  be  collected  on  a  global  basis,  assembled 
at  a  central  point  at  the  highest  national  levels,  and  decisions  made 
and  implemented,  all  in  a  few  minutes  at  most!  However,  even  in 
more  restricted  military  operations  (e.  g.  ,  "limited  war")  the  areas 
involved  are  likely  to  be  more  extensive,  certainly  the  necessary 
reaction  time  will  be  less,  and  decisions  must  be  made  at  higher 
organizational  levels  than  in  the  past. 

In  the  tense  world  situation,  with  several  countries  already 
in  possession  of  nuclear  armament  and  others  nearly  so,  there  has 
been  a  re-awakening  of  interest  in  "controlled  disarmament"  and 
"arms  control."  Again,  a  detailed  analysis  of  the  information-eval¬ 
uation-decision  requirements  for  effective  and  reliable  "inspection" 
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and  "arms  control"  is  beyond  the  scope  of  this  report.  It  is  interest¬ 
ing  to  note,  however,  that  they  bear  a  striking  resemblance  in  many 
respects  to  the  similar  needs  for  effective  control  of  actual  military 
ope  rations. 

Surveillance  is  an  information  processing  and  control  function. 
In  a  basic  sense  it  is  similar  to  any  "management"  or  "control"  sys¬ 
tem.  There  are  four  essential  elements  in  any  control  system: 

1.  Some  established  goal  or  objective. 

2.  Some  means  of  obtaining  information  about 

the  situation. 

3.  Evaluation  of  the  situation  against  some 

criteria  (derived  from  the  goal  or  objective), 
and  determination  of  what  action  should  be 
taken,  within  available  resources,  to  modify 
the  situation  to  achieve  the  objective. 

4.  Implementation  of  the  necessary  action. 

Considerable  work  has  been  done  on  outlining  the  general 
characteristics  of  probable  future  operations  from  the  specific  view¬ 
point  of  surveillance,  in  defining  the  general  parameters  of  the  sur¬ 
veillance  function  in  these  classes  of  operational  situations,  and  in 
isolating  the  critical  areas  and  limits  or  constraints  imposed  by  the 
nature  of  the  operational  problems.  This  work  is  reported  elsewhere 
(19  ,20)  and  will  not  be  discussed  here  in  detail.  The  following  brief 
comments  on  the  four  essential  elements  of  the  surveillance  function 
derive  largely  from  these  studies. 
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Objectives  or  Goals.  It  is  fundamental  (though  often  not 
clearly  understood)  that  military  operations,  or  the  ability  to  conduct 
them,  exist  as  an  element  of  our  foreign  policy.  The  only  reason  for 
defense  and  military  preparations,  and  for  actual  military  operations 
as  well,  is  to  strengthen  thenegotiating  arm  of  our  diplomatic  rep¬ 
resentatives  starting  with  the  President.  Thus,  the  objective  or  goal 
in  any  surveillance  system  must  derive  from  our  national  policy. 

Of  course,  this  does  not  mean  the  personnel  engaged  in  res¬ 
earch  and  development  must  be  involved  in  the  formulation  or  even 
the  detailed  study  of  national  policy.  However,  they  must  understand 
that  this  relationship  exists  and  consider  the  effects  of  major  elements 
of  our  national  policy  on  the  function  of  surveillance.  For  example, 
as  discussed  earlier,  it  is  only  recently  that  the  nature  and  tempo  of 
military  operations  have  changed  so  as  to  demand  more  and  more 
detailed  control  to  be  exerted  at  the  higher  command  levels.  The 
increased  range  and  speed  of  delivery  systems  coupled  with  the  vast 
destructive  potential  of  the  weapons  themselves  has  dictated  a  closer 
interaction  between  military  control  systems  (such  as  surveillance) 
and  national  objectives  and  strategies. 

In  part  because  of  this  close  interaction  with  national  objec¬ 
tives  and  strategies,  more  and  more  attention  has  been  given  to 
"peacetime"  or  pre -hostilities  information  processing  problems. 

This  has  led  to  the  concept  of  deterrence  through  the  threat  of  immed¬ 
iate  massive  retaliation.  Thus,  we  have  the  problems  of  mainten¬ 
ance  and  control  of  a  large  peacetime  force  in  being,  with  a  large 
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part  of  this  force  in  a  high  state  of  readiness,  and  detailed  pre- 

i 

planning  for  various  contingencies  to  permit  faster  response  and  de¬ 
creased  vulnerability.  Effective  deterrence  requires  a  demonstrated 
capability  to  prevail  if  war  occurs.  It  is  not  enough  to  have  the  wea¬ 
pons  and  delivery  systems,  we  must  also  have  the  capability  to 
determine  when  and  how  to  use  them  as  an  effective  element  of  our 
foreign  policy,  to  adjust  their  state  of  readiness,  to  shift  their  loca¬ 
tions  or  concentration,  to  modify  their  capability,  etc.  ,  as  well  as 
their  employment  in  an  operational  sense  if  the  occasion  demands. 

To  serve  properly  our  national  objectives  and  strategies,  we  need  a 
surveillance  capability  effective  in  peace,  in  cold  war,  and  in  actual 
hostilities  of  all  types,  in  limited  war  as  well  as  in  general  war.  We 
have  needs  for  information  collection  and  processing  on  a  scale  pre¬ 
viously  unheard  of,  and  we  need  a  capability  for  evaluation  and  deci¬ 
sion-making  of  the  greatest  flexibility  and  the  highest  levels  of  soph¬ 
istication. 

Collect  Information.  Surveillance  deals  with  aerospace  sit¬ 
uations.  Hence,  we  must  obtain  information  about  all  types  of  aero¬ 
space  targets:  aircraft,  ballistic  missile s .  satellites,  etc.  We  need  to 
obtain  such  information  over  much  larger  volumes  of  space  than  ever 
before,  and  we  need  to  obtain  it  quickly.  We  need  to  detect  such 
targets,  to  determine  the  type  or  classification  of  each,  to  determine 
what  they  are  doing,  what  they  are  capable  of  doing,  and  what  they 
intend  to  do ! 

Much  of  this  information  cannot  be  measured  or  sensed 


-8- 


directly,  but  must  be  calculated  or  inferred.  The  most  that  our 
sensors  can  do  is  to  measure  signals,  or  observe  effects  which  are 
indicative  of  physical  parameters.  Even  these  observations  will  be 
more  or  less  incomplete  and  inaccurate,  i.e.  ,  will  contain  varying 
amounts  of  noise.  The  problem  is  further  complicated  by  the  fact  that 
(at  least  in  many  situations)  we  will  be  unable  to  locate  our  sensors 
in  places  which  would  give  us  the  best  information.  Also,  due  to  the 
necessity  to  communicate  over  long  distances,  data  will  be  subject  to 
further  degradation,  both  in  timeliness  and  quality. 

Evaluate  Situation  and  Determine  Action.  Since  the  data  from 
our  sensors  are  incomplete  and  inaccurate,  we  cannot  know  what  the 
true  situation  is.  At  best  we  will  have  some  indications  or  cues  as  to 
certain  parameters  of  the  situation.  We  must  operate  on  these  "noisy" 
data,  and  fill  in  the  blanks  or  unknowns  to  create  the  best  representa¬ 
tion  of  the  actual  situation.  We  must  then  evaluate  this  admittedly 
inaccurate  picture  of  the  situation  and  determine  the  best  utilization 
of  our  available  resources. 

This  is  an  information  processing  problem,  and  a  major 
portion  of  the  Air  Force  research  program  in  information  processing 
is  directed  toward  improving  our  capability  in  areas  pertinent  to  this 
type  of  problem.  It  is  significant  that  many  of  the  functions  which 
must  be  performed  are  closely  analogous  to  human  processes,  so 
that  investigation  of  human  performance  may  provide  valuable  inputs 
in  our  efforts  to  express  these  functions  mathematically  and  to  design 
devices  to  perform  them  automatically.  In  fact  it  appears  probable 
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that  the  greatest  payoff  may  come  from  more  effective  integration 
of  the  high  speed,  large  storage  capabilities  of  machines  with  the 
judgment  of  human  operators.  This  is  particularly  true  when  con¬ 
sidering  the  high  level,  executive  control  function.  It  seems  clear 
that  this  process  will  not  be  delegated  to  a  mechanical  device,  no 
matter  how  sophisticated. 

Implement  Necessary  Action.  In  order  for  any  control  system 
to  function,  the  capability  must  exist  to  implement  the  necessary 
action.  Although  this  function  is  outside  the  realm  of  surveillance, 
it  is  necessary  to  know  what  action  can  be  taken  (i.e,  ,  the  available 
resources)  in  order  to  select  the  most  appropriate  action.  It  is  also 
necessary  to  observe  the  results  of  the  action  in  order  to  reassess  the 
situation.  Surveillance  is  a  dynamic  process,  which  continually 
observes  and  evaluates  the  situation,  and  initiates  new  actions  as 
appropriate. 

As  we  have  noted,  present  and  future  surveillance  operations 
require  information  processing  on  an  entirely  new  scale,  both  in 
quantity  and  in  sophistication.  While  the  overall  situations  are  more 
complex,  the  necessary  reaction  time  is  markedly  less,  and  the 
available  data  contain  errors  and  omissions.  In  many  cases,  signi¬ 
ficant  information  is  buried  in  noise,  has  been  altered  or  distorted 
by  the  environment  (or  by  processing),  or  is  deluged  by  quantities 
of  less  important  details.  Large  numbers  of  variables  must  be 
manipulated  simultaneously,  with  varying  weights  depending  on  the 
parameters  of  the  situation.  Clearly,  these  are  not  deterministic 
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processes,  but  are  statistical  or  probabilistic  in  nature.  Yet  most  of 
our  large  scale  computers  are  deterministic  devices. 

A  portion  of  our  current  research  program  is  investigating 
various  approaches  for  constructing  mathematical  models  and  evolving 
concepts  and  techniques  for  mechanizing  processes  of  this  type,  e.  g.  , 
research  in  mathematical  statistics,  game  theory,  utility  theory, 
queuing  theory,  and  decision  theory;  research  on  system  theory,  syn¬ 
thesis,  and  simulation;  research  on  adaptive  processes,  learning, 
pattern  recognition,  correlation  techniques,  etc. 

Many  of  these  functions  involve  processes  normally  associated 
with  human  intelligence,  such  as  learning,  reasoning,  recognition 
(or  perception),  inference,  etc.  Although  the  vast  quantities  of  data, 
and  the  fast  reaction  time  emphasize  the  need  for  automation,  the 
current  state  of  knowledge  does  not  permit  mechanization  of  such 
processes,  except  perhaps  in  very  simple  situations.  Since  man  does 
learn,  recognize,  reason,  etc.  ,  experimental  investigations  of  his 
performance  in  carefully  controlled  environments  can  provide  a  valua¬ 
ble  tool  in  better  understanding  the  basic  mechanisms  involved  in  these 
processes.  Such  data  will  complement  the  more  abstract  studies  in 
the  search  for  concepts  and  methods  for  more  sophisticated  automa¬ 
tion. 

At  the  higher  organizational  levels,  the  nature  of  the  problems 
and  the  potential  impact  of  the  decisions  dictate  the  continuing  need 
for  human  participation,  at  least  as  a  monitor  and  as  a  final  arbiter. 
Man's  role  in  these  processes  and  the  extent  and  nature  of  his 
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affiliation  with  a  machine  complex  for  optimum  performance  is 
relatively  unexplored,  particularly  in  complex,  time-constrained, 
multiple  alternative  situations,  under  conditions  of  high  stress,  vari¬ 
able  risks,  and  dynamic  environments.  This  type  of  problem  also  is 
amenable  to  experimental  investigation. 

This  particular  report  deals  with  the  portion  of  our  research 
program  concerned  with  experimental  investigations  of  man-machine 
performance  in  a  variety  of  situations  representative  of  the  problems 
outlined  above. 

A  simulation  facility  has  been  developed  at  AFCRL  (9,28)  with 
the  capability  of  simulating  a  wide  variety  of  situations  of  varying 
degrees  of  complexity.  Utilizing  this  facility,  a  series  of  experiments 
has  been  conducted  in  which  human  operators  were  required  to 
analyze  and  evaluate  data  representing  dynamic  operational  aerospace 
situations,  to  make  appropriate  decisions,  and  to  initiate  corrective 
action.  The  objectives  of  this  type  of  experimentation  may  be  divided 
into  two  general  classes: 

1.  Investigation  of  human  performance  in  such 

processes  as  recognition  (or  perception), 
learning,  reasoning,  inference,  etc.,  and 
derivation  of  mathematical  models  of  these 
processes. 

2.  Determination  of  fundamental  criteria  and 

parameters  of  man-machine  interrelation¬ 
ships  in  evaluation  and  decision-making  in 
complex  situations. 
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Initial  experiments  have  concentrated  on  the  latter  type  of 
problem.  Although  some  consideration  has  been  given  to  the  executive 
control  problem  at  the  highest  national  levels  (e.  g.  ,  Presidential 
level  decisions  in  the  general  war  case)  (19  ),  it  was  decided  to  exam¬ 
ine  experimentally  the  threat  evaluation  and  action  selection  function 
(i.  e.  ,  the  executive  control  function)  in  a  generalized  tactical  aero¬ 
space  environment.  There  were  several  reasons  for  this  decision. 

For  one  thing,  the  Presidential  level  is  involved  in  many  other  deci¬ 
sion  problems  not  directly  related  to  the  military  operations.  Also, 
it  is  much  simpler  (although  still  quite  difficult)  to  simulate  reasonable 
situations.  Then  too,  it  was  felt  that  many  of  the  aspects  of  the  prob¬ 
lems  are  reasonably  similar  such  that  considerable  extrapolation  of 
results  can  be  made  both  to  higher  and  lower  organizational  levfels. 

In  essence,  this  is  a  problem  of  making  optimum  decisions  in 
very  complex  situations,  based  on  uncertain  data,  and  under  conditions 
where  the  possible  choices  of  action  have  uncertain  results.  Many 
unforeseen  situations  must  be  handled,  and  frequently  the  "rules  of  the 
game"  are  known  only  in  a  general  sense,  so  that  an  effective  decision¬ 
making  strategy  cannot  be  prepared  in  advance.  The  information  pro¬ 
cessing  system  must  be  able  to  derive  dynamically  the  detailed  rules 
of  the  game,  to  adjust  automatically  its  performance  to  these  changing 
rules,  and  to  adapt  its  strategy  to  the  actual  situations  presented. 

Thus,  although  general  criteria  for  evaluating  the  situation  normally 
stem  from  the  "goals  or  objectives",  the  detailed  criteria  and  strategy 
must  be  developed  within  the  control  complex  as  the  operation  progresses. 
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Through  such  experiments  it  is  hoped  to  derive  information 
on  (or  at  least  gain  a  better  insight  into)  some  of  the  more  critical 
elements  of  evaluation  and  decision-making  in  complex  situations, 
for  example: 

1.  The  basic  parameters  of  the  decision  process  and  how 
they  are  related: 

(a)  Can  complex  decisions  be  reduced  to  definable 
parameters  which  can  be  varied  individually  for  study? 

(b)  Methods  for  reducing  complexity  of  decision 
making  alternatives;  e.  g.  ,  sequential  vs  parallel  decisions,  "natural" 
matrices  for  certain  classes  of  decisions,  etc. 

(c)  Effects  of  type  and  quality  of  input  data  on  decision 
processes;  e.g.  ,  effects  of  "noise"  or  inaccuracies  in  input  data; 
does  missing  information  pose  the  same  problems  as  uncertain  data; 
effects  of  delays  in  receiving  data  (i.  e.  ,  timeliness);  effects  of 
approximation  or  summarization  of  information  (i.  e.  ,  filtering),  etc. 

(d)  Effects  of  different  types  of  erroneous  decisions  on 
the  decision  processes;  e.  g. ,  if  "wrong"  decision  can  be  catastrophic, 
does  this  tend  to  inhibit  or  delay  making  any  decision;  is  there  a 
tendency  to  make  "easy"  decisions  first  even  if  they  are  relatively 
unimportant;  how  to  handle  important,  but  extremely  low  probability 
events,  etc. 

2.  The  critical  aspects  of  man-machine  interrelationships 
in  the  evaluation -decision  process: 

(a)  Effective  criteria  for  optimum  trade-offs  between 
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men  and  machines;  what  functions  must  man  perform  and  what 
functions  can  be  assigned  to  machines;  are  these  fixed,  or  do  they 
change  in  a  dynamic  situation,  etc. 

(b)  Essential  parameters  for  man  to  serve  effectively 
as  a  monitor  (i.  e.  ,  a  safety  check)  and  as  commander  (i.  e.  ,  final 
arbite  r). 

(c)  Effects  of  overloading;  e.  g.  ,  does  effectiveness  of 
man-machine  complex  collapse  suddenly  at  critical  overload,  or 
deteriorate  slowly;  can  the  complex  recover  effectively  as  overload 
decreases;  how  to  prevent  or  minimize  effects  of  overload,  etc. 

(d)  Criteria  for  altering  decision  strategies  and  dynamic 
revision  of  processing  procedures;  how  determined;  how  best  comm¬ 
unicated  to  machine  elements,  etc. 

(e)  Functional  specifications  for  displays  and  controls, 
e.g.  ,  how  best  to  present  a  situation  summary;  how  to  query 
machines;  essential  elements  of  common  man-machine  language. 

3.  Significant  parameters  of  human  performance  in  evalua¬ 
tion  and  decision-making: 

(a)  Procedures  utilized  by  man  in  such  processes  as 
recognition  (or  pe  rception),  learning,  reasoning,  inference,  etc.  Can 
these  procedures  be  expressed  as  mathematical  models;  can  they  be 
related  toexisting  theories,  e.g.,  decision  theory,  etc. 

(b)  Methods  for  handling  uncertainties,  missing  data, 
errors,  delayed  data,  etc. 

(c)  Limitations  imposed  by  human  characteristics  * 
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on  efforts  to  integrate  closely  man-machine  activities. 

(d)  Identification  of  personality  or  other  factors  which 
could  predict  "good"  or  "poor"  performance  as  a  decision-maker. 

(e)  Methods  for  testing  mathematical  models  or  auto¬ 
matic  devices,  vs  human  performance  in  decision-making. 

(f)  Effects  of  overloads,  stress,  high  risk,  etc.,  on 
human  decision-making. 

This  experimental  program  is  aimed  primarily  at  exploring 
the  basic  principles  of  the  general  problems  of  evaluation  and  deci¬ 
sion-making  in  complex  situations.  While  the  generality  of  the  prob¬ 
lem  has  been  maintained  to  the  maximum  extent  possible,  it  has 
been  necessary  to  select  more  specific  situations  for  simulation,  i..e.  , 
the  threat  evaluation  and  action  selection  function  in  a  generalized 
tactical  aerospace  environment.  Consequently,  it  is  expected  that 
certain  information  or  principles  of  system  design  and  operation, 
will  derive  automatically  which  may  find  application  in  more  immediate 
system  problems.  While  this  is  not  a  primary  goal  of  the  research, 
this  type  of  information  will  be  identified  and  reported  as  appropriate. 

Many  approaches  have  been  employed  to  investigate  tactical 
decision-making  as  related  to  field  and  operation  type  situations. 
Exercises  are  constantly  being  run  by  all  service  elements  to  evaluate 
specific  force  combinations  related  to  their  missions.  Human 
decision-making  as  related  to  air  surveillance  systems  (  31  *32)  has 
recently  been  surveyed  in  detail.  Story  (29)  has  recently  published 
a  report  on  the  general  problem  of  defining  system  performance 
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criteria.  Glaser  and  Wilson  (13)  have  proposed  a  mathematical 
model  for  use  In  tactical  decision-making  research. 

However,  in  spile  of  all  this  effort,  little  experimental  work 
has  been  accomplished  to  define  parameters  relevant  to  decision¬ 
making  as  it  relates  to  surveillance  techniques.  Mackworth  ( 17  ) 
appears  tc  be  one  of  Llie  firs,  co  investigate  this  area  with  work  in  the 
context  of  decision-making  problems  in  combat  air  operations  of  the 
British  Fleet.  Project  Cadillac  (  7  )  experimentally  investigated 
human  threat  evaluation  and  action  selection  performance  in  airborne 
CIC's.  The  Cornfield  Program  {  25  )  appears  to  be  the  first  investi¬ 
gation  of  threat  evaluation  and  ac  tion  selection'in  a  general  sense  with 
a  man-computer  combination.  In  Cornfield,  several  experiments  were 
run  comparing  human  performance,  computer  performance,  and  man- 
computer  interaction  for  processing  a  series  of  simulated  naval  air 
defense  problems.  Jc.st  prior  to  tnese  studies,  Chapman,  et  al  (  6  ) 
ran  a  series  of  experiments  evaluating  operational  air  defense  crews, 
but  the  emphasis  was  on  crew  performance,  rather  than  the  decision 
processes  involved.  Currently,  there  are  studies  underway  at  Rand 
Corporator,  System  Development  Corporation,  and  at  Applied  Physics 
Laboratory  exercising  the  command  function  in  terms  of  the  decision¬ 
making  invoiced,  but  in  each  case  the  results  are  specific  to  immediate 
systems. 

Adequate  experimental  data  on  human  command-decision 
behavior  are  still  virtually  non-existent.  In  attempts  to  fill  this  lack, 
the  use  of  an  aerospace  surveillance  simulator  seems  to  have 
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advantages  over  the  alternatives  of  logical  analysis,  single -variable 
laboratory  experiments,  and  field  exercises.  The  usefulness  of  these 
alternative  approaches  are  recognized,  but  still  they  are  less  attrac¬ 
tive  for  several  reasons.  Logical  analysis  is  constrained  by  the 
validity  of  the  underlying  assumptions  made  about  the  human  decision 
processes.  Solid  data  are  still  lacking  here.  These  may  be  supplied 
in  part,  by  single -variable  laboratory  experiments,  but  this  process 
is  slow  and  is  not  suited  to  the  detection  and  measurement  of  critical 
combinations  of  variables.  There  is  also  a  lack  of  "realism" 
associated  with  these  studies.  Field  studies  produce  data  in  terms  of 
a  realistic  environment.  They  are  difficult  to  control  and  assess, 
however,  in  that  trials  under  the  same  conditions  cannot  be  repeated, 
and  the  manipulation  of  a  range  of  experimental  variables  is  usually 
impossible. 

The  use  of  a  simulation  facility  permits  a  reasonably  realis¬ 
tic  environment  to  be  generated.  This  environment  can  be  controlled 
to  a  degree  not  possible  in  the  field.  Thus,  repeated  trials  under 
similar  conditions  and  sufficient  data  points  can  be  obtained  to  insure 
stable  measures  of  the  processes  under  consideration. 

Feasibility  of  simulating  tactical  decision-making  problems 
with  the  equipment  used  here  was  investigated  by  Doughty  (9), 

Other  studies  were  conducted  to  implement  the  capability  of  the  facility 
(12,  28). 

The  experiment  reported  here  is  the  first  major  experiment 
in  a  series  designed  to  study  man-machine  performance  in  aerospace 
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command  and  control  tasks  cf  threat  evaluation  ard  action  selection 
(TEAS).  This  experiment  was  concerned  with  the  study  of  the  effects 
of  overall  task  lead  open  tne  performance  of  "tactical  decision-makers" 
under  circumsta*  ces  in  which  the  information  supplied  was  essentially 
complete  and  correct.  In  addition,  the  threats  which  they  were 
required  to  evaluate  and  counteract  were  cf  a  conventional  air-breath¬ 
ing  sort.  The  major  task  of  the  "commanders"  was  the  selection, 
from  an  inventory  which  varied  in  kind  and  number,  of  an  appropriate 
choice  of  counter  weapons,  with  the  outcome  of  any  action  more  or 
less  uncertain.  Actual  evaluation  of  threat  in  relative  or  ab  solute 
terms  was  crly  a  small  part  cf  the  job.  The  first  major  experiment 
was  intended,  in  part,  to  establish  some  "base-lines"  of  human 
decision-maki-g  performance  which,  as  noted  in  considering  other 
,  were  rot  available  from  previous  research. 


sire  die  s 


SUMMARY 


This  experiment  was  the  first  major  study  in  a  series 
concerned  with  decision  making  in  simulated,  complex  aerospace 
surveillance  situations.  Nine  experienced  commanders  in  two  groups 
were  required  to  evaluate  the  threat  of  and  select  counter-actions 
against  a  total  of  over  9,  000  pre-programmed  aircraft  tracks  in  120 
experimental  problem  sessions  plus  a  somewhat  larger  number  of 
training  sessions.  Commanders  were  provided  with  a  geographic 
situation  display  and  auxiliary  displays  of  identifying  data  as  well  as 
weapon  status  information.  The  objectives  of  this  experiment  were  to 
establish  some  base-lines  of  human  capabilities  and  limitations  in 
complex,  dynamic,  real-time  and  realistic  military  decision  making 
and  to  begin  to  describe  the  processes  as  well  as  to  establish  some 
criteria  for  man-machine  combinations. 

Commanders  were  required  to  develop  the  tactics  necessary 
to  prevent  damage  by  hostile  weapons,  to  deplete  enemy  capability  by 
destruction  of  hostile  weapons  and  to  conserve  counter -forces  as  far 
as  possible.  They  were  faced  with  from  60  to  96  tracks  in  the 
surveillance  situation  and  were  provided  with  a  variety  of  aircraft  and 
missile  interceptor  weapons.  The  counter  actions  the  commanders 
selected  were  evaluated  and  implemented  and  the  outcomes  were  made 
known  to  them.  The  effectiveness  of  the  actions  selected  was  com¬ 
pared  to  both  their  potential  effectiveness  and  to  a  sample  of  indepen¬ 
dent  and'logical"  actions  such  as  might  be  arrived  at  by  various 
"idealized"  decision  making  systems. 

Summary  measures  of  the  commanders'  performance  are 
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p re sented  below.  Figure  1  summarizes  damage  assessed  in  terms  of 
per  cent  of  weapons  lost  by  load  and  threat  complexity  level  for  the 
two  experimental  groups,  and  for  cost  in  terms  of  weapons  expended. 

It  can  be  seen  that  Group  I  (the  commanders  with  the  most  experimen¬ 
tal  experience)  was  generally  superior  in  performance  in  having 
sustained  less  damage  than  Group  II.  The  latter  group,  however, 
showed  considerable  improvement  with  limited  practice  except  at  the 
highest  loads.  Lower  damage,  in  the  long  run,  can  only  be  achieved 
by  a  greater  use  of  the  weapons  available.  This  often  requires  the 
commitment  of  large  numbers  of  weapons  and  little  or  no  delay  in 
assignment.  The  fact  that  Group  I  assigned  more  weapons  accounts 
largely  for  their  superior  prevention  of  damage. 

Another  performance  measure  which  indicates  success  in 
selecting  actions  is  indicated  by  the  number  of  kills  achieved,  as 
reported  in  Figure  2.  If  damage  alone  were  considered,  an  incomplete 
picture  of  performance  would  result  since  elimination  of  a  small 
number  of  critical  tracks  could  minimize  damage.  Overall  efficiency 
in  the  tasks  required  here  also  calls  for  a  reduction  of  enemy  capa¬ 
bility.  In  performance  of  this  task  it  may  be  seen  that  Group  I  was 
more  effective  in  dealing  with  the  threat  situation.  While  Group  II 
shows  an  ability  to  "keep  up"  with  the  load,  they  showed  more  evidence 
of  becoming  saturated  at  the  highest  loads  than  the  more  experienced 
group. 

Figure  3  and  4  compare  the  actual  commanders'  performance 
with  the  summary  performance  of  three  classes  of  "analytical"  or 
logical  solutions  to  the  same  decision  making  problems.  The  better 
"idealized"  solutions  showed  somewhat  better  damage  prevention 
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than  the  real  commanders  and  killed  substantially  more  of  the  enemy, 
but  at  the  expense  of  nearly  total  exhaustion  of  the  counter -weapon 
resources. 

The  following  were  the  overall  findings  of  this  experiment: 

1.  Substantial  damage  was  prevented  by  the  experimental 
commanders  with  the  more  experienced  group  showing  superior 
performance. 

2.  Weapons  were  well  preserved  in  terms  of  maintaining 
a  capable  posture. 

3.  Reusable  threat  vehicles  were  destroyed  to  the  extent 
that  little  capability  remained. 

4.  Successful  strategies  were  derived  by  the  commanders, 
but  ability  to  verbalize  the  approach  was  not  fully  developed. 

5.  Track  load  did  not  provide  the  deleterious  effect 
expected;  in  fact,  the  most  experienced  group  showed  little  evidence 
of  leveling  off  in  performance.  In  addition: 

(a)  A  definite  pacing  effort  was  observed,  in  that,  as 
load  increased  the  commanders  increased  the  rate  at  which  they 
reacted  to  the  situation. 

(b)  Ability  to  sort  among  threat  and  weapon  categories 
did  not  level  off  in  this  experiment. 

6.  Frequent  exposure  to  near-saturation  task  loads  tended 
to  insure  optimum  performance  of  the  man-machine  combinations  in 
threat  evaluation  and  action  selection. 
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-23- 


95 


3  Ideal  "machine" 


T  rack  Load 


Figure  3,  Analytical  vs,  sample  empirical  performance: 
Threats  destroyed  at  two  levels  of  load. 
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Figure  4.  Analytical  vs.  sample  empirical  performance: 

Damage  and  weapons  used  at  two  levels  of  load. 
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PROCEDURES 


The  basic  apparatus  has  been  described  in  detail  in  previous 
reports  {  16,28  ).  The  schematic  site  layout  (Figure  5)  and  the  infor¬ 
mation  flow  diagram  (Figure  6)  provide  the  essential  elements  for 
understanding  this  study.  From  the  position  of  the  tactical  decision¬ 
maker,  the  environment  consisted  of  a  "command  post"  type  of 
facility.  The  decision-maker  was  seated  at  a  digital  display  console 
on  which  was  displayed  the  geographic  position  situation  of  the  aero¬ 
space  forces  (both  enemy  and  friendly)  under  surveillance.  "Targets" 
or  tracks  were  indicated  by  spots  of  light  showing  on  a  cathode  ray 
tube  screen.  A  small  portion  of  track  history  was  shown  by  the  per¬ 
sistence  of  previous  track  positions.  The  system  cycling  time,  i.  e.  , 
the  time  to  update  all  information  stored  on  the  drum  to  the  digital 
display,  was  10  milliseconds,  allowing  all  targets  to  be  viewed  without 
interruption.  Track  categories  were  capable  of  being  coded  for  IFF 
by  dimming,  brightening,  focussing  or  defocussing  the  track  spots. 
Track  categories  could  be  selected  for  display  or  inhibited  by  selec¬ 
tion  of  suitable  switches.  New  tracks  would  "blink"  on  entry  into  the 
situation.  This  blinking  could  be  inhibited,  at  time  of  interrogation 
with  the  photo-electric  light  gun,  by  depressing  a  small  thumb  switch. 
Tracks  could  be  allowed  to  remain  blinking  as  a  memory  aid  to 
indicate  that  further  action  was  requirtflor  action  should  be  delayed. 
Tracks  would  reinitiate  blinking  when  identification  or  other  ancillary 
data  changed.  Any  track  could  be  selectively  called  up  by  track 
number,  and  when  the  particular  track  number  was  inserted,  that 
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track  blip  would  blink  to  show  its  location,  while  all  others  would  be 
inhibited  from  blinking.  Upon  such  a  track  interrogation,  all  ancillary 
information  associated  with  the  track  (IFF,  speed,  altitude,  category, 
and  personal  identity  coding)  would  be  read  out  in  alphanumeric  form 
on  a  small  display  panel  located  on  the  top  of  the  digital  display. 

The  basic  track  position  situation  and  identifying  data  were  supplied  to 
the  system  from  digital  recordings  on  magnetic  tape,  routed  through  a 
selective  device  by  which  different  track  block  combinations  could  be 
displayed.  Additional  details  may  be  found  in  Appendices  I  and  II. 

In  addition  to  the  electronically  generated  displays  described 
above,  the  decision-maker  had  other  display  sources  to  consider  for 
the  solution  of  each  problem.  The  CRT  had  a  skeleton  diagram  overlaid 
on  it,  showing  the  location  of  the  friendly  weapon  sites.  This  bounded  a 
surveillance  area  of  300  by  300  nautical  miles.  In  addition,  two  weapon 
status  boards  were  located  directly  behind  the  digital  display  console 
(a  schematic  drawing  of  which  is  in  Appendix  I).  The  status  display 
boards  in  this  experiment  were  manually  posted  by  two  airman  techni¬ 
cians  behind  the  edge -lit  lucite  board.  An  action  log  was  maintained 
for  the  decision-maker  by  another  airman  technician  who  recorded  each 
action  as  it  occurred  and  made  this  information  available  on  call  to  the 
commander.  In  addition  the  airman  technician  notified  the  commander 
immediately  of  priority  outcomes  (missed-intercept  data).  The  remain¬ 
der  of  the  equipment  mainly  consisted  of  interior  communication  links 
(a  modified  AN /GT A -6),  pencils  and  paper,  and  a  digital  clock. 

The  overall  operating  procedure  is  best  described  by  describing 
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portions  of  an  operation  step  by  step  in  time  sequence. 

1.  The  tape  recorded  track  situation  data  were  fed  into  the 
digital  communications  system,  thence  to  a  drum  store,  and  thence  to 
the  situation  display.  The  first  targets  appeared  at  problem  time  zero. 
Weapon  status  at  the  onset  of  each  mission  is  shown  in  Appendix  I. 

2.  The  tactical  commanders  first  duty  was  to  interrogate  the 
first  tracks  one  at  a  time.  On  the  basis  of  position,  identification, 
kind,  speed,  etc.  ,  he  would  act  against  each  target. 

3.  The  action  selected  consisted  of  the  track  number  to  be 
attacked,  the  weapon  kind,  the  site  from  which  it  was  to  be  drawn,  the 
armament  type  and  the  number  of  such  weapons.  The  mode  of  employ¬ 
ment  was  assumed  to  be  that  of  a  single  flight  group.  There  were  no 
grouped  "raids"  or  raid  assignment  capabilities,  as  such,  in  this 
experiment. 

4.  The  action  selected  was  simultaneously  recorded,  together 
with  the  time  it  was  selected,  by  the  commanders'  airman  technician 
and  by  the  "scramble  clerk"  at  the  referee  station. 

5.  The  scramble  clerk  passed  the  slip  on  which  he  had  record¬ 
ed  the  action  data  to  the  inventory  clerk.  Appropriate  deductions  from 
inventory  were  communicated  to  the  weapon  status  board  where  the 
board  keepers  made  the  necessary  changes. 

6.  The  inventory  clerk  passed  the  assignment  to  one  of  three 
action  referees  (one  for  missile  assignments  and  the  other  two  for 
aircraft  assignments).  The  referee  then  evaluated  the  action  against 
a  scoring  sheet  containing  the  distributions  of  "kills"  ar.d  "misses" 


-29- 


(for  each  track)  along  with  outcome  time  data.  A  sample  referee  sheet 
with  detailed  explanations  may  be  found  in  Appendix  II. 

7.  The  action  outcome,  if  a  kill,  was  passed  to  the  closeout 
technician  who  would: 

(a)  cause  the  track  in  question  to  be  removed  from  the 
system  at  the  specified  time  (this  time  was  also  preplanned  and  was  a 
function  of  the  intercept  time  which  in  turn  depended  upon  the  speed  of 
the  weapon  assigned  and  the  distance -to -go  at  scramble  time)  and, 

(b)  transmit  the  outcome  information  to  the  commander 
(via  the  commanders  technician)  at  the  outcome  time. 

8.  The  outcome,  if  not  a  kill,  was  also  transmitted  to  the 
commander  (the  commanders'  technician  gave  the  commander  the  out¬ 
come  as  a  priority  message)  at  the  outcome  time,  but  without  closing 
the  track  out  of  the  system. 

9.  Each  action  referee  also  had  a  running  record,  by  time,  of 
all  preprogrammed  potential  damage  which  could  be  inflicted  by  any 
track.  If  a  track  capable  of  inflicting  damage  was  killed  prior  to 
causing  the  damage,  no  losses  were  assessed.  Otherwise,  the  poten¬ 
tial  weapon  site  was  assessed  for  possible  damage,  and  losses,  if  any, 
were  posted  on  the  weapon  status  boards.  Such  weapons  lost  were 
removed  permanently  from  the  inventory  of  that  site. 

Experimental  missions  had  a  duration  of  45  minutes  for  all  track 
situations  presented.  The  details  of  the  experimental  plan  may  be  found 
in  Appendix  II.  The  commanders  (or  tactical  decision-makers)  in  this 
experiment  were  all  First  Lieutenants  and  Captains  with  extensive 
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ACItW  experience  in  operational  sites  and  in  the  Experimental  Sage 
Sector  as  well.  They  were  briefed  in  detail  on  the  purposes  of  the 
experiment  (except  for  such  knowledge  of  experimental  conditions  as 
would  vitiate  the  results  if  foreknown)  and  on  their  duties  (cf  Appendix 
I).  They  were  also  given  an  intelligence  briefing  prior  to  each  mission 
and  the  opportunity  to  review  any  of  the  ground  rules  or  operating  con¬ 
ditions  or  procedures.  Numerous  practice  problems  were  run  with 
each  commander  prior  to  the  collection  of  the  data  described  below 
(cf  Appendix  II). 
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RESULTS  AND  DISCUSSION 


Since  this  research  focused  on  the  adequacy  of  decision  making 
by  executives,  the  primary  datum  was  the  record  of  action  taken  against 
each  of  the  total  of  approximately  9,400  tracks.  Specifically,  for  each 
track,  the  kind  of  action  taken,  the  time  it  was  selected,  and  the 
number  of  actions  were  all  recorded.  In  subsequent  analyses  of  these 
recordings,  two  particular  measures  of  decision  adequacy- -delay  in 
decision,  which  was  defined  as  time  from  track  appearance  to  action 
order,  and  correctness  of  decision,  defined  as  appropriateness  in 
application  of  action  criteria --we re  derived.  These  data  will  be 
reported  and  discussed  with  the  goal  of  specifying  the  adequacy  of 
decision  performance,  under  varying  conditions  of  load,  with  regard 
particularly  to  the  fourfold  task  that  had  been  given  the  executive. 

This  mission  was  the  classic  one  assigned  commanders  from  time 
immemorial:  (1)  to  defend  his  forces  (minimize  damage  to  the  weapon 
areas);  (2)  to  damage  the  enemy  (maximize  number  of  threat  vehicles 
destroyed);  (3)  to  conserve  counter  weapons  so  as  to  consume  a  mini¬ 
mum  of  forces  consistent  with  objectives  (1)  and  (2),  above;  and  (4)  to 
develop  his  own  strategy  under  the  constraints  imposed  by  the  ground 
rules  (cf  Appendix  I  for  details). 

From  the  post-mission  debriefing,  additional  records  were 
made.  Specific  items  toward  which  this  inquiry  was  directed  were 
whether  or  not  he,  the  commander,  was  aware  of  his  success  or 
failure  in  coping  with  each  mission,  whether  or  not  he  could  evaluate 
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the  adequacy  of  the  advance  intelligence -briefing,  and  his  evaluation 
of  the  adequacy  of  his  weapon  inventory. 

The  first  results  to  be  presented,  below,  specify  for  the  ex¬ 
perimental  groups  the  average  rates  of  events  occurring  versus  actions 
selected,  outcomes  achieved,  commanders'  errors  in  data  processing 
and  commanders'  awareness  of  their  performance.  The  subject's 
results  are  also  contrasted  against  analytical  measures  of  perform¬ 
ance.  Finally,  comparisons  of  individual  subject' s  performance 
against  critical  features  internal  to  each  experimental  problem  are 
summarized  in  order  to  consider  implications  for  decision  making  as 
a  process. 

I.  GROUP  PERFORMANCE  MEASURES 

A.  Analysis  of  weapon  selection  performance. 

In  the  following  graphs,  direct  comparisons  between  the  two 
experimental  groups  by  track  load,  threat  level,  and  alternative  sitt 
configurations,  for  the  experimental  variables  under  consideration, 
are  shown.  Grotp  I  performance  usually  is  indicated  by  the  top  and 
middle  sets  of  graphs;  the  lower  set  of  the  two  indicates  the  more 
complex  threat  level.  The  bottom  set  of  graphs  shows  the  Group  II, 
or  less  experienced  group,  performance  for  both  levels  of  threat 
complexity. 

1.  Threat  processing 

Figure  7  depicts  the  total  numbers  of  tracks  which  had 
entered  the  situation  display  at  the  end  of  each  five  minutes  period, 
the  total  numbers  against  which  at  least  one  action  had  been  taken  and 
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the  numbers  against  which  no  action  had  been  taken.  These  data  are 
plotted  as  functions  of  the  load  and  general  threat  l  evel  for  GroupII 
plus  the  defensive  site  configuration  for  Group  II  plus  the  defensive 
site  configuration  for  Group  I  commanders.  Each  plot  represents  the 
number  of  tracks  of  a  class  still  present  in  the  situation.  The  plots 
of  "tracks  remaining"  include  both  processed  (i.  e.  ,  those  acted 
against)  and  unprocessed  tracks.  The  between  the  "input 

tracks"  and  "tracks  remaining"  reflects  the  number  of  kills  or 
successful  selected  actions.  Particularly  worthy  of  note  in  this  figure 
is  the  difference  in  performance  between  the  main  experimental  group 
(I),  more  experienced,  and  the  second  group,  (II),  that  received  the 
eight-site  configurations  only.  This  difference  can  be  seen  particularly 
for  kill  rate  achieved,  and  also  in  the  disparity  of  the  number  of  tracks 
not  processed.  The  backlog  of  unassigned  tracks  increases  with  load 
and  is  much  flatter  over  time  for  Group  I.  The  less  experienced 
group  as  might  be  expected,  allows  it  to  build-up  faster  and  remain 
longer.  There  is,  however,  very  little  difference  in  either  kill  rate 
or  backlog  between  five  and  eight  sites,  though  both  seem  most  fre¬ 
quently  lower  for  five  sites.  Further,  threat  level  II  seems  to  affect 
Group  II  much  more  severely  than  Group  I  (as  might  be  expected  due 
to  amount  of  practice).  In  interpreting  these  figures  it  should  be  re¬ 
called  that  load  built  up  to  a  peak  at  the  twenty  to  twenty -five  minute 
interval  and  then  diminished  slowly  until  the  end  of  each  problem  at 
fcrty-five  minutes.  No  track  close-out  of  any  value  was  achieved 
before  problem  time  ten;  this  was  due  to  the  tendency  to  assign  on  the 
initial  tracks  with  fighter-interceptors,  as  will  be  shown  later. 
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2.  Weapon  assignment  rate 

The  rates  at  which  both  missiles  and  aircraft  interceptors 
were  assigned  are  indicated,  below,  in  Figure  8.  Both  experimental 
groups  reacted  with  an  increasing  rate  of  response  as  load  increased, 
with  Group  I  showing  the  higher  response  rate. 

The  Commanders  were  instructed  to  derive  their  own  solutions 
to  these  problems,  and  this  figure  indicates  very  strongly  that  the 
approach  that  was  adopted  by  both  groups  was  an  increasing  use  of 
fighter  interceptors  early  in  each  problem,  and  the  use  of  missiles 
later  when  problem  load  was  high.  The  quicker  feedback  time  of  the 
missiles  permitted  this  weapon  to  be  used  as  backup,  i.  e.  ,  to  elimin¬ 
ate  targets  where  an  initial  aircraft  assignment  had  failed,  or  to  kill 
quickly  any  target  appearing  during  the  later  stages  of  the  problem, 
which  constituted  the  "loaded"  interval.  Note  that  the  assignment  behavior 
of  Group  II  does  not  show  much  early  economy  of  missiles  except  as 
load  96.  This  figure  and  the  indicated  kill  rates  from  Figure  7  indi¬ 
cate  that  essentially  the  problems  were  dealt  with  in  the  first  35 
minutes  or  so  of  problem  life,  with  an  adequacy  such  that  few  assign¬ 
ments  remained  to  be  generated  at  the  end  of  each  problem.  Assign¬ 
ment  of  weapons  from  five,  as  opposed  to  eight,  sites  appear  to  be 
accomplished  equally  well,  with  the  exception  of  the  highest  two  loads 
at  threat  level  two  for  Group  I.  Here  a  slight  difference  is  noted  for 
a  short  interval  of  time  in  aircraft  assignments  and  also  in  missiles, 
but  at  only  the  highest  track  load. 

Two  interpretations  may  be  advanced  to  account  for  the  increase 
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in  the  assignment  rate  under  increasing  threat  load:  (1)  An  increased 
rate  of  response  due  to  increased  awareness  of  the  threat,  or  (2)  a 
pacing  effect  of  the  stimulus  inputs  either  due  to  the  way  the  problems 
were  designed  or  to  some  general  property  of  behavior  in  the  face  of 
increasing  load  situations.  The  data  presented  in  Figure  9  appears 
to  support  choice  of  the  latter  interpretation. 

3.  Weapon  assignment  rate  by  threat  class 

Figure  9,  below,  shows  the  rates  at  which  weapons  were 
assigned  against  the  three  threat  categories  capable  of  inflicting 
damage:  bomber,  fighter-bomber  and  fighter  inputs.  Only  data  on 
Group  I,  for  threat  level  two,  will  be  discussed  here.  This  figure 
supports  a  pacing -by -load  concept,  i.  e. ,  the  assignment  rate  increases 
as  a  direct  function  of  load.  It  is  notable  that  this  assignment  rate  is 
highest  for  fighter-type  threats,  indicating  that  the  commanders  were 
reacting  more  to  increase  in  number  of  tracks  than  to  specific 
discriminations  of  the  extent  of  the  threat  implied  by  a  particular 

track.  The  second  group  of  commanders  was  impeded  more  by  the 

.that  .  ,  ,  , 

increasing  load,  in/their  rate  increases  was  consideraly  lower, 

(these  data  are  not  shown  here). 

High  assignment  rates  on  fighters  are  due,  in  part,  to  the 
nature  of  their  entry  into  the  problems.  Because  they  are  small 
vehicles,  and  hence,  would  present  less  frequent  radar  returns,  many 
were  programmed  to  appear  for  the  first  time  near  a  weapon  site.  The 
experimenters  were  able  to  observe  that  when  load  was  high  many 
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subjects  tended  to  give  priority  to  threat  tracks  that  were  approaching 
and  friendly  sites.  Fighers  appearing  here  may  have  had,  in  effect, 
a  nuisance  value  that  forced  the  commander  to  react  against  them. 

This  will  be  examined  more  fully  under  the  discussion  of  delay  times, 
below.  With  regard  to  assigning  fromfive  versus  from  eight  sites, 
the  figure  indicates,  again,  about  equal  facility  on  the  part  of  the 
commanders. 

4.  Weapon  assignment  against  "real11  vs,  "apparent"  threat 
While  Figure  8,  above,  indicated  the  weapon  assignment 
rates  against  perceived  threat  in  general,  Figure  10  compares  the 
weapon  assignments  against  the  "reals",  or  tracks  with  damage 
potential,  versus  weapon  assignments  against  the  "apparent"  threats, 
that  look  threatening  due  to  their  labels  (e.g.  ,  Foe  Bomber)  and  the 
general  direction  that  they  are  heading,  but  which  do  not  possess 
actual  damage  potential.  The  uppermost  set  of  graphs  indicates  the 
"real"  threat  input  rate  (solid  line)  and  the  "apparent"  threat  (dotted 
line).  The  middle  set  of  graphs  indicates  the  assignment  rate  for 
missiles  (solid  line)  and  aircraft  (dotted  line),  for  both  site  modes 
(dot  equals  five -site,  open  circle  equals  eight-site),  for  Group  I.  The 
bottom  set  of  graphs  indicates  the  comparable  assignment  rates  for 
Group  II.  The  graphs  on  the  left  side  of  each  page  represent  threat 
level  I,  and  those  on  the  right  side  threat  level  II.  Note  that  the 
assignment  rate  for  "apparent"  threat  is  considerably  higher  than 
for  "real"  threat  tracks.  There  are  actually  more  of  the  former 
class,  and  the  ratio  between  apparent  and  real  threats 
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and  that  between  the  assignments  against  them  are  about  the  same. 

The  evidence  of  pacing  by  "number  of  tracks"  for  both  groups 
again  is  apparent.  Group  II,  however,  again  shows  some  increased 
delay  in  processing  all  tracks. 

The  fact  that  a  track  enters  the  situation  initiates  a  new  cycle 
in  the  command  function.  Each  such  entering  track  first  must  be 
evaluated  for  possible  threat.  This  evaluation  is  not  automatic;  one 
reason  for  this  being  that  the  target  of  ahostile  track  may  not  be 
obvious  for  quite  some  amount  of  track  life.  Assignments  must  be 
committed  early,  but  the  outcome  at  this  time  is  uncertain.  Neither 
man  nor  a  computer  acting  in  man's  place  can  go  beyond  an  extrapola¬ 
tion  based  on  past  history  of  the  displayed  hostile  track  when  it  comes 
to  prediction  of  the  probable  target,  if  the  rule  is  that  the  decision  is 
to  be  based  solely  on  track  data.  Where  little  additional  information 
is  presented,  the  decision  maker  must  assume  that  all  threats  are 
valid,  thus,  he  must  take  appropriate  action  on  all  of  them. 

5.  Weapons  assigned  per  kill 

If,  over  a  series  of  simulated  battles,  the  commander 
developed  skill  in  action  selection,  one  would  expect  to  see  him  in¬ 
creasing  the  efficiency  of  employment  of  his  weaponry.  Since  the 
action  outcomes  are  probabilistic  in  nature  (ranging  from  .  1  to  .999), 
the  number  of  weapons  to  achieve  a  kill  must  average  more  than  one. 
Figure  11  indicates  the  number  of  weapons  assigned,  including  some 
that  have  been  mis -matches  or  too  late  assignments  for  each  hostile 
track  killed.  The  uppermost  graphs  present  data  for  threat  level  I 
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and  indicates  that  more  weapons  were  assigned  atthis  level  than  at 
level  II.  Both  experimental  groups  tend  to  perform  alike  in  that  they 
regress  toward  a  three-on-one  weapon  assignment  per  kill. 

6.  Weapon  usage  rate  bv  interceptor  class 

The  commander  had  available  for  assignment  four  levels 
of  interceptor  capability  plus  several  varities  of  armament.  Hence, 
the  data  may  be  analyzed  to  determine  whether  or  not  each  interceptor 
performance  level  was  utilized  systematically  throughout  the  experi¬ 
ment.  Further,  it  may  be  asked,  do  these  rates  of  assignment  of  the 
alternative  interceptors  change  as  a  function  of  problem  complexity 
and  load  level?  Finally,  did  the  commanders  make  best  use  of 
lowest  performance  level  interceptors,  (red  class)  which,  because  of 
their  location  at  forward  and  vulnerable  sites,  should  be  utilized 
rapidly,  before  the  liklihood  of  their  being  lost  to  enemy  action  becomes 
too  great?  Figuresl2a  and  12b  summarize  the  use  of  these  different 
performance  classes  by  Group  I  and  II,  respectively. 

It  is  apparent  from  the  figures  that  the  missiles  are  used  exten¬ 
sively  by  both  groups,  but  Group  II  was  more  prodigal  with  this 
weapon  class.  In  fact,  the  extremely  high  missile  use  at  the  high¬ 
est  loads  by  both  groups  at  both  levels  is  further  evidence  for  a 
"pressure -of -load"  hypothesis.  Group  I  tended  to  use  the  missiles 
at  a  faster  rate  during  the  heavy  load  portion  of  the  problems,  espec¬ 
ially  for  the  five-site  configurations.  The  red  class  aircraft  were 
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employed  as  early  as  possible,  taking  into  account  that  the  matching 
criteria  must  be  met  by  having  appropriate  tracks  available,  to  obtain 
most  of  the  capability  from  this  class.  The  second  dip  in  this  function 
at  the  highest  load  indicates  that  many  fighers  returned  from  the 
initial  mission  early  enough  to  go  through  the  rearming  and  refueling 
cycle  and  be  assigned  again.  Blue  cla&3  interceptors  were  used 
steadily  throughout  the  missions  as  they  were  intended  to  be  used 
(they  could  stay  airborne  and  had  ample  stores  to  run  repeated  inter¬ 
cepts),  but  the  Green  class  interceptors  tend  to  be  used  less  frequently 
While  they  could  kill  the  highest  performance  threat,  apparently  the 
commander  desired  the  shorter  feedback  time  associated  with  missiles 
Both  groups  of  commanders  tended  to  expend  their  missiles,  while 
retaining  a  higher  percentage  of  fighter -interceptors. 

7.  Average  intercept  times 

As  a  sub-task  the  commander  was  instructed  to  achieve 
intercepts  or  kills  as  quickly  as  possible,  consistent  with  what  he 
believed  to  be  an  appropriate  action  strategy.  Following  this  dictum 
would  tend  to  reduce  the  overall  work  load,  and  give  the  decision 
maker  more  time  to  cnnsider  additional  necessary  actions,  both  of  the 
initial  and  reassignment  types.  Particularly,  the  commander  would 
have  more  time  to  consider  additional  threats  with  a  view  toward  iden¬ 
tifying  tracks  suitable  for  intercept  by  Red  and  Green  class  fighters. 
Early  assignments  of  these  fighters  was  specially  important  though 
not,  of  course,  at  the  expense  of  appropriate  assignment,  because 
timeliness  would  permit  RTB,  rearming  and  refueling,  and  an  early 
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return  to  active  inventory.  If,  however,  these  fighters  were  scrambled 
on  remote  tracks,  a  minimum  intercept  time  could  not  be  achieved, 
and  they  wou||  not  be  able  to  go  through  the  refitting  cy*le  in  time  to 
be  employed  again  during  the  mission. 

Figure  13  indicates  the  average  intercept  time,  cumulated  over 
each  five -minute  interval,  for  both  experimental  groups.  It  is  to  be 
noted  that  there  were  no  kills  achieved  during  the  first  five-minutes  of 
problem  time  and  usually  none  were  achieved  during  the  second  five- 
minutes  earlier.  Group  I  achieved  about  the  same  intercept  time  distri¬ 
butions  across  the  various  loads,  except  at  the  level  of  96  tracks, 
where  a  very  slight  increase  in  time  is  shown.  There  are  no  marked 
differences  shown  for  Group  I  between  different  threat  levels  or  alterna¬ 
tive  Site  configurations.  The  trend  toward  shorter  intercept  times 
later  in  the  problem  probably  was  due  to  the  fact  that  incoming,  most 
tracks  were  much  nearer  the  sites  from  which  weapons  were  scrambled 
by  the  time  intercepts  were  achieved.  Group  II  shows,  in  general,  the 
same  trends.  Those  commanders  appear  to  have  selected  shorter 
intercept  times,  but  this  may  have  been  due  to  inability  to  assign  as 
rapidly  as  did  members  of  Group  I.  (This  point  will  be  discussed 
further  under  action  delay  times).  Group  II  appears  to  have  achieved 
slightly  shorter  intercept  times  for  fighter  assignments  against  threat 
level  I,  than  was  the  case  for  threat  level  II. 

8.  Weapon  selection  time  delays 

As  the  overall  load  on  the  commander  increases,  we 
would  expect  increasing  delays  in  responding  to  the  threat  inputs. 
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Under  heavy  load  conditions  some  threats  should  be  missed  (not  react¬ 
ed  to  at  all)  and  others  should  be  reacted  to  only  after  considerable 
delay.  Tables  la  and  lb,  below,  summarize  the  median  delay  time 
(average  of  subject  medians)  and  gives  the  range  of  these  medians  for 
all  conditions  of  load,  threat  level,  site  configuration,  and  subject 
group  for  the  first  action  and  for  the  first  reassignment,  if  any, against 
each  track. 

The  median  response  time  remained  about  the  same  for  all  load 
conditions,  for  a  given  level  of  threat,  for  Group  I  data.  The  overall 
median  delay  for  Group  I  for  threat  level  II  was,  however,  ZO  seconds 
greater  than  it  was  for  threat  Level  I.  Similarly,  the  delay  difference 
for  Group  I  subjects  for  the  first  reassignment,  threat  level  I  versus 
threat  level  II,  was  only  3  seconds  (1:27  for  threat  level  I  vs.  1:30  for 
threat  level  II).  To  put  these  delays  in  proper  perspective,  it  must  be 
pointed  out  that  the  fact  that  the  delays  shown  do  not  increase  seriously 
at  the  heaviest  (96)  track  loads  indicates  a  considerable  increase 
(lower  loads  were  presented  first)  in  ability  to  select  actions,  since 
at  this  level  over  50  per  cent  more  tracks  are  available  for  processing 
than  at  the  initial  problem  level  (60  tracks). 

Delays  in  acting  against  threat  level  I  bomber  tracks  are  some¬ 
what  longer  (39  seconds)  than  the  delay  for  all  tracks  processed  (1:46 
vs.  2:25).  For  threat  level  II  bomber  tracks,  the  difference  in  delay 
is  41  seconds  (2:06  vs.  2:47).  This  increase  in  delay  may  be  explained 
as  partly  due  to  the  fact  that  usually  the  bombers  enter  the  problems 
from  the  remote  edge  of  the  situation  display.  Thus,  initially,  they 
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TABLE  la 


Median  Delay  Times  for  Initial  Assignments  Against 
Threat  by  Experimental  Conditions 


INITIAL  ACTION  Track  Load 

Row 


Group  I 

60 

72 

84 

96 

Mean 

Threat 

All 

1:37 

1:46 

1:46 

1:51 

1:46 

Level 

Bomber 

2:00 

2:46 

2:39 

2:16 

2:25 

I 

F  ighter-Bomber 

1:43 

1:15 

2:05 

1:55 

1:55 

F  ighter 

1:32 

1:11 

1:19 

1:38 

1:25 

Threat 

All 

2:25 

2:'l? 

1:42 

2:02 

2:06 

Level 

Bomber 

2:51 

2:40 

2:18 

3:18 

2:47 

II 

Fighter -Bomber 

1:24 

3:10 

1:44 

2:35 

2:18 

Fighter 

1:52 

1:55 

1:29 

1:34 

1:43 

Group  II 

Threat 

All 

3:11 

3:45 

3:56 

4:31 

3:51 

Level 

Bomber 

3:48 

10.  20 

7:34 

8:26 

7:32 

I 

Fighter-Bomber 

3:10 

1:42 

5:59 

3:18 

3:32 

F  ighte  r 

2:01 

1:35 

1:44 

1:49 

1:47 

Threat 

All 

4:08 

6:09 

4:31 

5:44 

5:10 

Level 

Bomber 

5:45 

5:43 

8:26 

12:09 

8:01 

II 

Fighter -Bomber 

6:30 

13:03 

3:18 

5:48 

7:10 

Fighter 

2:59 

1:59 

1:49 

5:58 

3:11 
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TABLE  lb 


Median  Delay  Times  for  Second  Assignment  Against 
Threat  by  Experimental  Conditions 


SECOND  ACTION 


Group  I 

60 

Threat 

All 

1:10 

Level 

Bomber 

1:01 

I 

F  ighte  r -Bombe  r 

0:56 

Fighter 

1:05 

Threat 

All 

1:19 

Level 

Bomber 

1:38 

II 

Fighter -Bomber 

2:15 

Fighter 

1:10 

.Group  fl 

Threat 

All 

0:35 

Level 

Bomber 

0:50 

I 

Fighter -Bomber 

0:35 

Fighter 

0:41 

Threat 

All 

0:55 

Level 

Bomber 

0:50 

II 

Fighter -Bomber 

1:00 

F ighte r 

1:00 

Track  Load 


72 

84 

96 

Row 

Mean 

1:10 

1:43 

1:44 

1:27 

1:19 

1:41 

1:52 

1:28 

1:10 

2:17 

1:38 

1:30 

1:01 

1:34 

1:28 

1:17 

1 

24 

1:18 

2:05 

1:30 

1 

41 

1:34 

2:34 

1:52 

1 

57 

1:52 

1:58 

2:00 

1 

15 

1:21 

1:41 

1:22 

1:12 

1:19 

1:13 

1:05 

1:11 

1:14 

1:07 

1:05 

1:58 

0:58 

1:44 

1:18 

1:06 

1:00 

1:12 

1:00 

1:10 

1:18 

1:18 

1:05 

0:55 

1:07 

1:15 

1:02 

1:16 

1:44 

1:10 

1:18 

1:13 

1:12 

1:12 

1:09 

are  located  far  from  the  defended  sites.  Hence,  the  obtained  delays 
support  the  observation  that  tracks  near  the  sites  were  processed 
more  rapidly.  Median  delays  for  second  action  on  bomber  tracks 
averaged  1:28  for  threat  level  I,  and  1:52  for  threat  level  II,  indicating 
a  slight  increase  which  may  be  due  to  the  increased  complexity  of 
threat  level  II.  The  difference  between  the  median  delay  for  all  second 
actions  and  median  second  actioi)  delay  for  bombers  alone  by  Group  I 
is  only  3  seconds  (1:27  for  all  second  action  delays  vs.  1:30  for  re¬ 
assignment  on  bombers).  For  threat  level  II  the  difference  between 
the  median  delay  for  all  second  actions  and  second  actions  on  bombers 
is  22  seconds  (1:30  for  all  second  action  delays  vs.  1:52  for  bomber 
second  action  delays),  again  in  the  direction  of  more  delay  on  bomber 
tracks. 

Tending  to  counterbalance  the  longer  delays  in  acting  against 
bombers,  delays  on  fighters  were  slightly  shorter  for  Group  I  when 
compared  to  the  median  first  action  delay  for  all  tracks  (1:25  vs.  1:46 
for  threat  level  I,  and  1:43  vs.  2:06  for  threat  level  II).  Again  con¬ 
sidering  Group  I  alone,  on  the  second  action  there  was  no  marked 
effect  of  threat  type  (delay  was  1:27  for  all  tracks  vs.  a  range  of  1:17 
to  1:30  for  all  classes  in  threat  level  I,  and  delay  was  1:30  for  all 
tracks  vs.  a  range  of  1:22  to  2:00  for  all  classes  in  threat  level  II). 

The  second  group  (II)  delayed  longer,  in  general,  in  initiating 
all  first  actions  than  did  Group  I.  However,  they  reassigned  for 
the  second  actions  with  a  slightly  shorter  delay  than  did  Group  I 
(1:05  for  Group  II  vs.  1:27  for  Group  I  for  threat  level  I,  and  1:05  for 
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Group  II  vs.  1:30  for  Group  I  for  threat  level  II).  This  difference  may 
be  explained  in  part  by  the  previously  described  tendency  of  Group  II 
to  concentrate  primarily  on  the  tracks  adjacent  to  the  weapon  sites, 
as  indicated  when  the  delays  for  the  specific  threats  - -bombe  rs  and 
fighter-bombers--are  considered.  These  subjects  (Group  II)  were 
apparently  preoccupied  with  preventing  damage  to  their  sites. 

To  achieve  optimum  solutions  to  these  simulated  battle  problems, 
attention  must  be  given  to  each  track  as  soon  after  track  initiation  as 
possible.  The  purpose  of  the  long  range  missile,  in  part,  was  to 
permit  the  commander  to  take  potential  high  threats  out  of  the  prob¬ 
lem  at  the  earliest  possible  time  and  at  greatest  distances,  thereby 
reducing  not  only  the  threat  potential  but  also  the  subsequent  workload. 

9.  Weapon  posture  at  termination  of  each  mission 

One  aspect  of  the  commander's  assigned  task  was  to 
retain  a  capable  weapon  posture  while  sustaining  minimal  damage, 
and  destroying  as  many  threat  vehicles  as  possible.  Thus,  number 
of  weapons  remaining  at  the  end  of  each  mission  provides  another 
indication  of  how  well  the  commander  processed  each  load  level. 

Table  2,  below,  summarizes  the  average  number  of  weapons  remain¬ 
ing  at  the  end  of  each  mission  by  load,  threat  level,  site  configurations, 
and  subject  groups.  In  these  data  we  can  see  that  both  subject  groups 
were  quite  similar  in  performance.  Load  shows  its  effect  in  that  the 
percentage  of  weapons  remaining  falls  off  as  load  increases.  Group 
II  retained  more  weapons  at  the  highest  load  level,  but  this  may  be 


-56- 


Weapon  Capability  Remaining  at  End  of  Missii 
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because  those  commanders  were  unable  to  make  as  many  assignments 
at  this  level  of  load  as  were  made  by  Group  I  commanders.  For 
Group  I,  differences  in  residual  weapons  between  threat  levels  and 
between  site  configurations  were  very  small. 

B.  Probabilistic  outcome  sampling  efficiency. 

In  real  battles  action  outcomes  are  probabilistic  in  nature 
and  will  undoubtedly  remain  so,  primarily  due  to  the  complexity  of 
the  conditions  associated  with  them.  To  simulate  this  aspect  of  the 
executive  function  a  probability  distribution  ranging  from  zero  chance 
of  success  for  a  single  action  to  near  certainty  (.999--  3  salvos  of  3 
missiles  in  short  range  mode)  was  generated  for  this  experiment  (cf 
Appendix  II).  This  distribution  was  used  to  provide  a  wide  variety  of 
success  or  failure  values  associated  with  the  alternative  action  choices 
put  before  the  commander.  It  is  believed  that  the  ability  to  assess 
accurately  and  rapidly  the  conditions  and  implications  of  actions,  as 
they  are  conditioned  by  changing  outcome  values,  will  be  a  prime 
requisite  of  the  executive  decision-making  role  in  future  surveillance 
systems. 

In  this  study  the  probability  value  associated  with  each  action 
outcome  is  a  more  sensitive  indication  of  the  value  or  magnitude  of 
the  action  selected  than  is  the  fact  that  the  action  consisted  of,  say, 
assigning  three  aircraft  instead  of  one  to  a  track.  Under  certain 
circumstances,  for  example,  three  Red  aircraft  can  have  a  probability 
of  successful  outcome  as  low  as  .  5  or  as  high  as  .  8.  Figure  14,  below, 
summarizes  the  selection  of  outcome  probabilities  by  load,  threat 
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Number  of  Actions 


60  Tracks 


Threat  Level  1 


72  Tracks 


Probability  Value 

Figure  14.  Action  outcome  (kill)  probability  sampling 
distributions. 
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GROUP  I 


Aircraft  0 - -  8 -site 

• - *  5-site 

Missiles  O - 8-site 

e - 5 -site 


GROUP  n 


^ - Missiles,  Level  U 

^ -  Aircraft,  Level  II 

O - Missiles  Level  I 

O  — 1  -  -  -■  Aircyaft  Level  I 


level,  site  configurations,  weapon  class  (missile  or  fighter -intercep¬ 
tor)  and  commander  groups.  Each  point  represents  the  number  of 
times  that  each  probability  value  was  sampled.  Derivation  of  the  pop¬ 
ulation  universe  is  a  nearly  impossible  task  considering  the  number  or 
weapons  available  and  their  possible  combinations.  Missiles  alone, 
for  example,  may  be  drawn  one,  two,  or  three  at  a  time,  and  in  the 
short  range  mode,  even  up  to  nine  may  be  assigned  to  a  single  action. 
With  60  items  in  the  inventory,  however,  only  a  few  such  draws  could 
be  made  before  the  inventory  was  exhausted.  One  single  function  des¬ 
criptive  of  missile  kill  probability  sampling,  i.  e.  ,  showing  the  mean 
potency  of  the  commander's  action  choiC’  s,  could  be  generated,  but 
it  would  have  to  be  based  on  average  values  from  many  combinations 
of  possible  missile  sampling  rates,  polled  across  load  conditions, 
subject  groups,  etc. 

Figure  14  presents  several  aspects  of  how  the  commanders 
Bampled  the  probability  values.  The  modal  point  for  sampling  prob¬ 
abilities,  i.e.  ,  the  most  commonly  selected  potency  of  weapon  assign¬ 
ment,  for  aircraft  ranges  from  .  5  to  .  6  (for  both  levels  of  threat,  both 
groups)  and  shows  no  decrease  with  increasing  load.  Such  a  decrease 
might  have  been  expected  on  the  basis  that  increased  load  might  pro¬ 
duce  increased  delay  in  assigning,  increased  rate  of  inappropriate 
assignments,  and  other  decrements  in  action  selection  performance. 
Any  such  decreases  in  decision  effectiveness  would,  of  course,  be 
reflected  in  lower  kill  probabilities,  since  the  probability  is  based  on 
consideration  of  the  several  dimensions  of  decision.  Except  for  Group 
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I  at  the  highest  load  and  second  level  of  threat,  no  changes  are  appar¬ 
ent;  there,  the  eight-site  sampling  for  aircraft  remains  at  .  5,  while 
the  five -site  sampling  has  increased  to  .  6,  Also  to  be  noted  is  the  fact 
that  slightly  more  items  are  selected  at  the  middle  values  in  the  five- 
site  mode  (Group  I). 

Probability  sampling  for  the  missiles  shows  a  tendency  toward 
bimodal  distribution.  This  trend  begins  to  drop  out  at  the  highest  load 
for  Group  I  and  quite  definitely  stops  for  Group  II.  Group  II  comman¬ 
ders  also  tended  to  select  more  of  the  higher  probability  missile  com¬ 
binations. 

Group  I  shows  essentially  the  same  sampling  for  the  five  vs.  the 
eight- site  configurations,  except  that,  in  general,  more  actions  of 
each  value  were  selected  for  threat  level  U,  the  problems  with  a 
larger  actual  number  of  threatening  hostile  tracks.  For  Group  II  through¬ 
out  there  is  a  tendency  to  employ  proportionately  more  aircraft  than 
missiles  for  threat  level  I  and  to  reverse  this  preference  for  threat 
level  II. 

C.  Damage  Assessment 

At  first  consideration  it  would  seem  that  damage  sustained 
would  provide  a  good  indication  of  how  well  a  surveillance  and  control 
system  had  functioned.  However,  prevention  of  damage  to  his  weapon 
sites  is  but  one  aspect  of  the  overall  task  the  commander  was  given. 

Also,  sustaining  damage  does  not  indicate  necessarily  that  the  decision¬ 
making  complex  functioned  inadequately.  Rather,  damage  may  in 
many  cases  be  due  to  the  binary  nature  of  the  outcome  of  the  particular 
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actions  chosen.  For  any  one  sample  set  of  actions,  high  probability 
outcome  choices  may,  due  to  chance,  show  a  relatively  higher  or 
lower  number  of  kills.  This  research  is  more  concerned  with  whether 
the  commander  adjusted  to  the  result  of  the  damage  assessment  and 
still  functioned  effectively.  In  some  cases,  the  commander  may  have 
had  to  accept  a  lesser  damage  to  prevent  a  greater  damage;  for  exam¬ 
ple,  to  accept  fighter  penetrations  while  reserving  weapons  to  kill 
bombers  and  other  high  priority  threats. 

1.  Total  damage  assessed 

The  average  total  damage  assessed  for  all  experimen¬ 
tal  conditions  is  indicated,  below,  in  Figure  15.  The  upper  set  of 
graphs  indicates  total  damage  incurred  by  Group  I  at  different  loads. 
The  circle  and  solid  line  entry  indicates  threat  level  I  and  eight- site 
damage;  the  circle  and  dotted  line  indicate  threat  level  I  andfive-site 
damage.  Within  this  portion  of  the  figure,  the  upper  solid  line  and 
dot  indicate  threat  level  II  and  eight-site  damage,  while  the  dotted 
line  and  the  dot  indicates  five -site  damage.  The  lower  portion  of  the 
figure  indicates  the  damage  incurred  by  Group  II.  The  dotted  line 
indicates  the  damage  for  threat  level  I,  and  the  solid  line  threat  level 
II.  Inspection  of  these  figures  shows  that  Group  II  received  somewhat 
more  damage  across  threat  levels,  and  notably  more  damage  at  load 
96  for  threat  level  II.  These  commanders  received  less  practice  at 
each  load  level  and  thus  had  less  chance  to  adjust  to  each  level  of 
load  and  develop  a  strategy  to  cope  with  the  load  to  the  same  extent 
that  Group  I  did.  Hence,  more  damage  under  conditions  of  maximum 
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Figure  15.  Total  damage  assessed. 
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problem  difficulty  might  have  been  expected,  and  in  fact  it  was  found. 

The  commanders  in  Group  I  were  cautioned  that  the  five -site 
problems  might  be  more  critical  for  damage,  due  to  the  fact  that  the 
same  total  weapon  inventory  was  stored  in  fewer  locations.  Hence, 
a  hit  would  have  the  potential  of  destroying  a  larger  share  of  the  wea¬ 
pons.  Figure  16  summarizes  the  damage  received  for  the  sites  that 
were  common  to  both  the  five -site  and  the  eight- site  configurations. 

The  forward  airfield  (site  number  one)  is  represented  by  the  upper¬ 
most  set  of  graphs.  This  site  received  the  greater  amount  of  total 
damage,  particularly  when  it  appeared  in  the  five-site  mode,  for 
threat  level  II.  The  next  lower  set  of  graphs  describes  the  damage 
received  by  airfield  three.  Here,  it  may  be  seen  that  both  threat 
levels  caused  almost  no  damage  except  for  the  highest  load  level  in 
the  five -site  configuration.  The  set  of  graphs  next  to  the  bottom  set 
summarizes  damage  to  airfield  four.  For  this  field  threat  level  I 
produced  about  the  same  damage  for  all  loads  and  for  both  site  config¬ 
urations.  Threat  level  H,  which  included  tracks  with  a  great  total 
capacity  to  damage,  caused  more  damage  in  general.  The  eight-site 
configuration  showed  more  damage  forthe  first  three  loads,  but  this 
graph  crosses  below  the  function  representing  the  five-site  rate  at  the 
highest  load  level.  The  lowest  set  of  graphs  stands  for  missile  site  B. 
It  is  apparent,  here,  that  very  few  penetrations  were  permitted  in  this 
weapon  area  although  several  were  possible.  In  the  five-site  config¬ 
uration,  damage  was  caused  only  at  the  highest  load  level  and  threat 
level  H. 

The  previous  figure  indicates  that,  as  a  rule,  damage  was 
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confined  to  the  forward  area  sites.  Of  additional  interest  is  the  prob¬ 
lem  time  when  damage  began  to  occur.  On  the  basis  of  the  problem 
program  one  bomber  penetration  could  have  occurred  as  early  as  five- 
minutes  from  problem  onset.  Figure  17  summarizes  the  damage  assess¬ 
ment  as  a  function  of  problem  time. 

The  left-hand  set  of  graphs  summarizes  the  damage  for  Group 
I.  Load  is  indicated  downward  from  the  top  of  the  page.  In  general, 
the  differences  between  site  configurations  are  negligible  until  the 
terminal  portion  of  load  96.  The  early  damage  indicated  for  the  eight- 
site,  threat  level  II  problem  for  loads  of  72  and  84  was  due  to  the  early 
bomber  penetrations  mentioned  above.  This  threatening  target  was 
usually  assigned  with  the  appropriate  matching  fighter,  but  this  fighter 
could  not  complete  the  intercept  before  bomb  drop  time.  The  graph 
for  this  condition  (eight-site,  threat  level  II)  for  load  96  shows  that  the 
commanders  were  able  to  eliminate  this  kind  of  threat  if  they  hit  on  the 
plan  of  an  appropriate  use  of  missiles. 

Turning  to  the  performance  of  Group  II  (right  hand  page),  we 
can  see  that  the  less  experienced  commanders  were  unable  to  eliminate 
this  particular  threat  (early  bomber).  The  reduced  scale  of  the  ordin¬ 
ate  is  intended  to  permit  a  relative  comparison  between  the  two  groups. 
As  in  previous  summaries  it  may  be  seen  that  threat  level  II  caused, 
in  general,  more  damage  for  this  group.  Again,  as  with  Group  I,  the 
Group  II  executives  were  able  to  prevent  most  of  the  damage  for  the 
first  twenty -five  minutes  of  the  problems.  It  is  to  be  recalled  that  the 
load  within  each  problem  also  peaked  shortly  after  this  time,  and  the 
effects  of  this  increase  in  the  internal  load  are  apparent  in  the  damage 
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sustained,  (cf,  Appendix  I  for  potential  penetrations  cumulated  over 
time). 

It  is  to  be  noted  that  on  only  one  occasion  did  a  site  (Red  air¬ 
field  two)  have  to  be  closed  down  due  to  two  successful  bomber  pene¬ 
trations,  although  at  several  sites  as  many  as  four  or  five  bomber 
penetrations  were  possible.  When  this  over-kill  potential  is  consid¬ 
ered,  it  is  apparent  that  a  creditable  performance  was  turned  in  by 
both  commander  groups.  A  consideration  of  the  analytic  solutions 
(cf  Table  8)  also  indicates  this  fact.  Only  rarely  does  an  analytic 
solution  indicate  no  damage  whatsoever. 

D.  Data  Processing  Efficiency. 

1.  Hostile  tracks  not  processed 

This  section  is  intended  to  show  how  adequately  various 
aspects  of  the  presented  data  were  considered  by  the  experimental 
commanders.  It  has  been  shown  previously  (Figure  10)  that  the 
commanders  could  not  discriminate  the  "real"  threats  in  the  time  they 
had  to  act,  with  the  kind  of  information  that  was  presented.  Also, 
there  was  no  indication  that,  as  load  increased  and  as  they  received 
additional  exposures  to  the  program,  they  were  able  to  single  out 
certain  aspects  of  the  threat  for  special  treatment  or  consideration. 

In  fact,  all  tracks  were  reacted  to  as  real  threats,  and  in  some  cases 
as  with  actions  against  the  fighter  inputs,  the  actions  selected  were 
greater  than  necessary  against  the  particular  threat  class. 

Table  3,  below,  summarizes  the  hostile  tracks  not  processed 
under  different  conditions  of  load  and  damage  potential.  It  can  be  seen 
that  some  potentially  damaging  tracks  are  missed  by  both  groups 
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throughout  the  experiment.  Although  with  practice  both  groips  showed 
considerable  improvement  in  processing  the  increasing  load,  Group 
It,  with  less  experience,  was  still  considerably  below  the  main  group 
in  performance  as  judged  by  this  index.  The  success  of  Group  II  in 
coping  with  the  real  threat  is  only  an  apparent  advantage  (cf  Figure 
13),  since  this  success  was  achieved  by  delaying  the  response  to  a 
track  until  the  intent  was  quite  obvious.  This  increased  delay  was  not 
purposeful --these  commanders  just  could  not  process  the  increasing 
load  at  any  faster  rate.  It  was  shown  previously,  that  their  reaction 
times  to  scramble  against  the  bomber  and  fighter -bomber  inputs  were 
somewhat  longer  than  the  delay  times  for  Group  I.  They  were  appar¬ 
ently  so  occupied  in  processing  tracks  that  were  near  the  sites  they 
could  not  attend  to  the  tracks  that  initiated  in  remote  areas  of  the 
situation  display.  Such  remote  tracks  should,  under  more  favorable 
load  conditions,  have  been  killed  before  their  threat  became  acute. 

The  increase  in  missed  tracks  at  the  72  track  load  appears  to 
be  a  practice  effect,  especially  for  Group  I,  as  the  higher  loads  show 
fewer  tracks  not  processed.  The  total  not  processed  for  Group  II 
for  threat  level  II  and  96  tracks  is  2  per  cent  of  the  total  tracks,  the 
total  for  Group  I  for  threat  level  II  and  96  tracks  is  1  per  cent  of  the 
tracks  not  processed.  The  commander  had  the  option  of  accepting  a 
possible  penetration  by  a  fighter,  which  could  inflict  only  a  single 
weapon  penalty.  Yet,  from  the  very  low  numbers  of  unprocessed 
tracks,  it  appears  that  they  did  not  employ  this  option,  but  rather 
attempted  to  continue  to  maximize  kills  throughout  the  entire  course 
of  the  experiment. 
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2.  Inappropriate  application  of  weapon  criteria 


To  make  effective  decisions,  i.  e.  ,  to  select  actions 
having  high  probabilities  of  kill,  the  commander  had  to  apply  correct¬ 
ly  the  matching  criteria  to  the  incoming  threat  tracks.  This  aspect 
of  his  task  consisted  mainly  of  selecting  a  weapon  that  could  match  or 
outperform  the  speed  and  altitude  of  the  incoming  track.  Also,  a 
scramble  site  had  to  be  selected  with  a  view  toward  achieving  a  short 
intercept  time  and  an  extra  margin  that  would  allow  the  intercept  even 
if  the  track  should  turn  away  to  make  the  action  into  a  tail  chase. 

(a)  Weapon  matching  with  target  tracks. 

Since  the  surveillance  data  were  presented  as  100 
per  cent  reliable,  and  since  the  weapon  inventory  was  tailored  to  the 
problem  load,  mis-matching  by  under  matching  could  be  due  only  to 
error  on  the  part  of  the  commander.  When  an  undermatch  occurred, 
it  might  have  been  due  to  a  momentary  confusion  of  the  commander 
such  that  he  mis-appraised  the  characteristics  of  the  threat  track. 

Of  course,  the  chance  cannot  be  ruled  out  that  when  faced  with  a 
depleted  weapon  inventory,  a  commander  might  permit  some  faint 
hope,  e.  g.  ,  a  change  in  threat  track  speed,  altitude  or  a  heading 
might  make  a  match  out  of  an  apparent  mis-match--to  influence  him 
to  order  into  action  a  weapon  of  inadequate  capability.  The  referee 
team  did  not  believe,  however,  that  this  "hope"  hypothesis  accounted 
for  any  substantial  number  of  actions. 

Table  4  summarizes  mis-match  performance.  We  can  see 
that  mis-matching  reached  a  peak  at  the  third  load  level  (84  tracks). 
After  that  load,  the  commander's  performance  began  to  improve 
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TABLE  4 

Distribution  of  Inappropriately  Assigned  Weapons  by  Threat 
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considerably.  The  eight-site  problem  appears  to  have  been  more 
difficult  but  performance  improved  here  also.  The  total  of  29  under¬ 
matches  out  of  1152  possible  tracks  at  load  96  and  threat  level  I  for 
Group  I  indicates  a  mis -match  error  of  approximately  two  and  one- 
half  per  cent. 

The  second  group  of  commanders  generated  fewer  actions,  but 
their  mis-match  :  i  rate  also  was  a  little  over  two  per  cent.  Out  of 
7116  foe  and  unknown  tracks  (cf  Appendix  II)  presented  to  Group  I,  they 
mis-assigned  on  60  bombers,  31  fighter-bombers,  and  114  fighters, 
for  a  total  of  2.  9  per  cent  error.  Group  II  mis-assigned  on  15  bombers, 
4  fighter-bombers,  and  26  fighters  of  a  total  of  1229  foes  and  unknowns 
for  a  total  of  2.  5  per  cent  error. 

(b)  Weapon  choices  resulting  in  tail  chases. 

At  the  moment  that  the  typical  action  choice  was  made, 
the  commander  could  not  judge  with  absolute  certainty  that  the  particu¬ 
lar  intercept  would  not  result  in  a  tail  chase.  The  commanders  in 
this  study  were  not  shown  weapon  track  data;  they  were  told  that  the 
weapons  were  monitored  by  a  transponder  re  porting -in  to  the  machine 
that  handled  the  intercept  computations,  and  they  were  told  further 
that  inappropriate  actions  would  be  signaled  to  the  commander  via  his 
technician.  The  reason  for  this  system  was  that  to  generate  and  make 
available  track  data  on  the  approximately  4,  000  intercepts  scored  for 
this  study  would  have  required  a  very  large  storage  capability  and  a 
large  facility  to  program  real-time  feedback.  It  is  doubtful  whether 
weapon  track  data  would  have  aided  the  commander  in  locating  inappro¬ 
priate  intercepts  in  advance  of  the  signal  fed-back  to  him,  due  to  the 
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heavy  track  loads  involved.  At  any  rate  no  commander  decommitted  in 
advance  of  any  inappropriate  action  signals.  In  fact,  this  is  apparently 
one  avenue  taken  by  the  executive  to  reduce  the  work  load--to  wait  for 
such  information  from  the  system. 

If  a  commander  waited  long  enough,  a  tail  chase  situation  would 
become  obvious;  with  a  tail  chase  established,  to  kill  would  require  a 
missile  or  a  manned  interceptor  with  a  large  speed  advantage.  The 
commanders  appear  to  have  performed  relatively  well  i  see  Table  5 
below.)  The  table  presents  absolute  numbers,  ranging  from  zero  tail 
chases  for  the  60  and  84  track  loads  (with  Group  1  commanders  in  the 
five-site  configuration)  to  a  high  of  twenty-three  tail  chases  for  the  9 6 
track  load  (with  the  same  commanders  in  the  eight-site  configuration). 
Interpretation  of  these  absolute  numbers  should  take  into  account  the 
larger  total  number  of  intercepts  at  higher  problem  loads  and  the  modi 
larger  number  of  problem  runs  with  Group  I  subjects.  When  such 
differences  in  all  entries  are  considered,  there  is  no  consistent  effect 
of  track  load  upon  rate  of  tail  chase  generation.  Second,  it  is  apparent 
that  there  were,  overall,  very  few  tail  chases.  All  conditions  pooled 
yields  about  two  per  cent  tail  chases. 

(c)  Distribution  of  action  choices  resulting  in  out-cf- 

range  intercepts. 

A  more  serious  error  than  tail  chase  generation  is 
commitment  of  a  weapon  to  an  impossible  intercept.  The  discrimination 
made  at  the  situation  display  to  evaluate  the  possibility  of  achieving 
intercept  before  the  target  track  goes  out-of-range  is  quite  similar  to 
the  judgment  of  tail  chase  probability  at  the  time  of  assignment.  When 
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an  out-of-range  scramble  had  been  ordered  the  commander  had  the 
opportunity  to  consider  the  track  for  10  minutes.  After  that  period  the 
system  signaled  that  the  outcome  would  be  negative  in  ten  more  min¬ 
utes.  As  stated  previously  for  tail  chase  assignments,  the  commander 
relied  heavily  on  this  feed-back  of  mission  outcome. 

The  frequency  distribution  of  actions  going  out-of-range  is 
indicated  in  Table  6,  below.  Reference  to  that  table  shows  that  Group 
I  selected  more  inappropriate  actions  in  this  regard  for  the  eight-site 
configuration,  and  also  selected  more,  in  general,  for  threat  level  I. 
Group  II  showed  no  marked  difference  by  threat  level,  and  neither 
group  showed  consistent  load  effects.  The  total  error  for  Group  I 
for  selection  of  out-of-range  intercepts  was  a  little  over  one  per  cent 
and  for  Group  II  it  was  about  three  per  cent. 

3.  Estimate  of  hostile  threat 

The  present  Study  was  concerned  primarily  with  action 
selection  performance,  and  for  that  reason  little  emphasis  was  placed 
upon  recording  direct  estimates  of  threat.  These  estimates  can  be 
derived  indirectly  from  the  data  by  considering  the  kind  and  amount 
of  each  action  employed  against  each  input  threat  class.  These  actions 
were  not  determined  by  threat  evaluation  alone,  but  varied  as  a  func¬ 
tion  of  the  inventory  available.  Hence,  they  do  not  indicate  accurately 
how  great  the  commander  considered  the  various  threats  to  be.  To 
get  around  this  source  of  inaccuracy,  the  commander  was  required  to 
state  an  assignment  priority,  numbered  from  one  to  four,  at  the  time 
of  action  selection.  In  the  case  of  the  missile  actions,  this  priority 
number  gave  the  commander  the  opportunity  to  decommit  or  downgrade 
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low-priority  assignments,  and  employ  the  weapons  on  higher  priority 
threats  without  entirely  passing  over  a  lesser  threat.  In  the  comman¬ 
ders  training  and  instructions,  threats  were  ordered  as  follows:  a  bomber 
was  considered  a  priority  one  target;  a  fighter-bomber,  priority  two;  , 

i 

a  fighter,  priority  three;  and  other  threat  categories  (cargos,  troop  j 
carriers,  and  reconnaissance),  priority  four.  A  bomber  departing 

j 

from  the  situation,  but  which  the  commander  still  desired  to  kill  so 
as  to  deny  hostile  capability,  could  |>e  indicated  as  priority  two  or  even 
three.  A  fighter -bomber  about  to  penetrate  a  site  should  be  called  a 
priority  one  target.  The  frequency  with  which  each  of  the  four  prior¬ 
ities  actually  was  assigned  in  the  study  is  presented  below,  in  Table  7. 
Reference  to  Table  6  shows  that  the  primary  order  was  followed 

!  i 

,  very  closely  except  in  the  case  of  tfye  fighter  input  s;  these  were  grossly 
overrated.  The  bombers  were  generally  called  "priority  one1',  with  j 

I  I 

Group  I  showing  a  tendency  to  report  many  of  them  as  "priori  tv  ;.wo."! 
Fighter -bomber s  were  generally  called  "priority  two"  but  both  groups 
showed  an  increasing  tendency  across  load  (with  two  exceptions)  to  call 
some  fighter-bombers  "priority  three."  The  fighers  were  labeled 
"priority  three"  for  the  most  part,  but  both  groups  labeled  some  of 
this  threat  class  "priority  two"  or  even  "priority  one".  This  would 
appear  to  support  previous  statements  that  fighters  were  acted  on 
somewhat  out  of  context.  Neithe  r  group  discriminated  "real"  from 
"apparent"  threats  in  their  priority  assignments. 

4.  Post-mission  debriefing  critique 

Immediately  after  each  experimental  run  the  comman¬ 
der  was  debriefed  on  the  following  items:  (a)  Commander's  ass  essment 


-78- 


I 


w 

CQ 

< 

H 


a 

3 

2 

o 

U 

4) 

a 

is 

o  -• 


Vi 

o 

c 

O 


S> 

XI 

•H 

4-1 
® 
•  H 

Q 


vD 

O' 


Hi  oo 

'O 

in 

o 

O' 

O' 

m 

o 

N 

eg 

v0 

o 

O' 

r~ 

og 

eg 

*81 

o 

rH 

o 

o 

rH 

© 

o 

rH 

in 

vO 

f\l 

• 

o 

• 

o 

o 

o 

o 

Hi  4» 

rH 

rH 

H 

Tf 

O 

CO 

o 

H« 

O' 

rH 

00 

in 

'O 

■* 

ol  " 

•* 

Osl 

o 

't 

o 

eg 

rH 

o 

co 

O' 

eg 

o 

o 

o 

o 

o 

a 

s0 

og 

rH 

o 

vO 

CO 

o 

o 

m 

in 

in 

© 

in 

og 

CO 

o 

a 

0 

NO 

rH 

o 

rH 

o 

00 

o 

o 

og 

00 

CM 

CM 

• 

o 

o 

o 

o 

o 

•* 

00 


HH 

CO 

oo 

© 

CO 

in 

^H 

rH 

o 

CM 

vO 

rH 

rH 

•# 

og 

CO 

a 

0 

og 

* 

• 

^H 

o 

co 

r- 

rH 

• 

rH 

• 

O 

NO 

og 

O 

• 

o 

• 

o 

o 

• 

o 

a 

a 

a 


,  C0 

vO 

00 

o 

NO 

og 

CO 

o 

CO 

nO 

O' 

o 

rt> 

in 

og 

© 

15. 

rH 

o 

o 

rH 

og 

rH 

o 

o 

rH 

r- 

CO 

rH 

o 

o 

o 

o 

o 

eg 

h- 


►H 

*** 

© 

m 

o 

o 

rH 

rH 

m 

CO 

o 

vO 

• 

a 

0 

20. 

o 

og 

m 

rH 

og 

o 

• 

N 

in 

© 

og 

o 

o 

© 

in  in 

•  • 

©  o 


oo  m  o 

•  •  • 

*-4  0  0 


O  CO 

•  • 

O  rp 


eg 

o 


o 

o 


*-*  o 

»  •  » 
o  o  o 


CO 


CM 


og 

© 

rH 

vO 

CO 

in 

o 

© 

rH 

rH 

in 

og 

rH 

CM 

© 

og 

in 

20. 

o 

• 

© 

© 

• 

© 

• 

o 

og 

CO 

rH 

og 

CO 

rH 

og 

CO 

Ui  o 


I 

O' 

1 


of  hia  performance  in  terms  of  success  in  coping  with  the  problem, 

(b)  Adequacy  of  the  weapon  inventory  for  each  problem,  (c)  Adequacy 
of  the  pre -mission  intelligence  briefing,  and  (d)  Commander's  modifi¬ 
cation  of  action  strategy  for  the  problem. 

The  results  of  the  analysis  of  debriefing  data  are  summarized 
in  Table  8,  below.  For  item  (a),  the  commanders  were  asked  to  rank 
their  performance  on  a  four-point  scale  ranging  from  believing  that 
they  were  ahead  of  the  problem  (in  terms  of  coping  with  it)  to  not  being 
able  to  rate  their  performance.  In  general,  it  appears  that  the  comm¬ 
anders  developed  a  reasonably  accurate  assessment  of  their  perform¬ 
ance.  It  is  to  be  noted  that  at  the  60  track  load  level,  only  8.  3  per  cent 
believed  that  they  were  ahead  of  the  problem.  This  estimate  rose  to 
29.  2  per  cent  at  load  72.  The  fact  that  half  the  commanders  thought 
that  they  were  at  least  abreast  of  this  problem  level  is  somewhat  at 
variance  with  data  reflecting  performance  in  avoiding  damage,  since 
damage  was  disproportionately  high  at  this  level.  However,  other 
measures  indicate  that  the  performance  was  about  the  same  as  on  other 
problems.  For  the  remaining  two  load  levels,  12.  5  per  cent  of  the 
commanders  reported  that  they  were  ahead  of  the  problem.  The  comm¬ 
anders  that  believed  that  they  were  keeping  up  with  the  load  increased 
from  37.6  per  cent  to  62.  5  per  cent  and  remained  there  for  the  highest 
load.  Commanders  who  reported  that  early  in  the  series  of  problems 
they  got  behind  tended  to  change  their  responses  with  increased  exper¬ 
ience  (20.  8  per  cent  for  load  60  to  zero  per  cent  for  loads  84  and  96). 
Finally,  a  large  residual  did  not  commit  themselves  (33.3  per  cent  for 
load  60  decreasing  to  25  per  cent  for  load  96). 
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When  the  commanders  were  asked  to  evaluate  the  weapon 
inventory,  reports  that  it  had  been  adequate  increased  from  50  per  ceqt 
for  load  60  to  70.  8  per  cent  for  load  96.  This  increase  may  be  taken  j 
as  an  indication  that  the  commanders  learned  to  utilize  the  weapons  | 
more  effectively,  since  inventory  stood  in  a  constant  ratio  to  load. 
Initially,  a  few  commanders  considered  the  weapon  inventory  inadequate 
(16.7  per  cent),  but  this  response  dropped  toward  zero  (4.2  per  cent)  | 

I 

at  load  96.  Also  expressed  initiallyj  was  the  feeling  that  more  weapons 
in  certain  categories  would  have  been  desirable.  This  response  tendejd 
to  drop  out  as  the  commanders  progressed  through  the  increasing  track 
loads. 

"Intelligence"  data  were  presented  to  provide  the  commanders  : 

I 

with  information  on  the  general  level  of  threat  to  be  expected  in  each 
problem.  Also,  they  were  instructed  to  consider  the  usage  of  each 
weapon  class  with  respect  to  this  intelligence  data.  It  was  pointed  out 

I 

to  the  commanders  that  threat  target  expectancies  varied  greatly 
from  problem  to  problem. 

A  large  proportion  of  the  commanders  rated  the  intelligence 
data  as  "good"  or  "adequate"  (58  per  cent  for  load  60  to  75.  0  per  cent 
for  load  96).  A  small  proportion  of  the  commanders  (8.3  per  cent  for 
load  60  to  4.  2  per  cent  for  load  96)  considered  the  intelligence  data  as 
unreliable.  With  experience  a  decreasing  proportion  of  the  comman¬ 
ders  indicated  no  evaluation  (33.  3  per  cent  for  load  60  to  16.  5  per  cent 
for  load  96). 

The  final  debriefing  question  was  concerned  with  the  comman¬ 
ders'  estimation  of  their  own  consistency  in  approach  (strategy)  to  the 
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solution  of  the  problems.  Part  of  the  overall  task  was  to  develop  a 
plan  of  solution  for  the  problems.  The  commanders  reported  that 
during  the  training  runs  they  tried  to  solve  the  problems  primarily  by 
utilizing  the  aircraft  interceptors,  while  saving  the  missiles  for  back¬ 
up.  It  is  to  be  noted,  here,  that  while  these  commanders  all  had  had  a 
great  deal  of  aircraft  control  experience,  none  of  them  had  observed 
the  control  of  missile  interceptors.  At  load  60  about  one -third  (37.  5 
per  cent)  stated  that  the  strategy  had  been  to  employ  aircraft  first, 
and  use  missiles  for  back-up.  Roughly  another  third  reported  chang¬ 
ing  strategy  on  the  60  load  problem,  and  the  remaining  third  of  the 
group  reported  no  evaluation  of  approach.  At  load  96  over  two -thirds 
(70.  8  per  cent)  reported  consistency  with  the  approach  reported  for 
training  (the  basic  use  of  aircraft  with  missile  back-up),  and  about  one 
fourth  (24  per  cent)  reported  no  evaluation.  Previous  measures  had 
indicated  an  increasing  use  of  missiles,  Earlier  in  each  problem,  to 
cope  with  the  increasing  load. 

5,  Post-mission  summary  data  feed-back  on  performance 
At  the  end  of  each  debriefing  session  each  commander 
was  given  his  performance  on  the  just  completed  mission  in  terms  of 
the  number  of  kills  achieved  by  Interceptor  class  (missiles  or  air¬ 
craft),  the  number  of  penetrations  sustained,  and  the  cost  in  terms  of 
weapons  used  per  kill.  This  information  was  provided  to  enable  the 
commander  to  validate  his  action  approach  (strategy)  by  assessing  the 
overall  effect  of  action  changes  which  he  might  have  introduced  during 
the  mission. 
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E.  Analytic  solutions  to  sample  problems. 


An  estimate  of  what  would  be  "good"  solutions  to  sample 
problems  was  desired  to  provide  a  standard  by  which  to  judge  the  per¬ 
formance  of  the  commanders  as  tactical  decision-makers.  These 
analytic  solutions  were  not  derived  within  a  game -theoretic  framework 
Rather,  it  was  felt  that  if  little  or  no  damage  was  incurred,  kills  were 
near  100  per  cent,  and  some  substantial  part  of  the  weapon  inventory 
remained  to  counter  other  threats,  adequate  defense  could  be  made  of 
the  analytic  solution  as  "good"  forthe  present  purposes.  The  solutions 
were  of  a  logic  or  problem-solving  type  in  that  specific,  related  rules 
had  to  be  observed  during  derivation  of  each  solution.  These  solutions 
would  be  expected  to  be  somewhat  above  the  actual  performance  of  the 
commanders,  since  the  latter  performed  in  a  time -constrained,  dyna¬ 
mic  situation,  whereas  the  "analytic"  or  "machine  solutions"  were 
derived  by  strict  rules  and  at  leisure. 

Three  levels  of  solution  were  derived:  (1)  idealized  "human" 
solutions  based  on  two  rates  of  assigning  actions,  (2)  idealized  "mach¬ 
ine"  solutions  where  delay  in  assigning  action  was  zero,  and  (3) 
random  "machine"  solutions  where  action  delay  was  zero  (as  in  (2), 
but  action  selections  were  drawn  at  random.  These  three  solutions 
were  derived  for  two  levels  of  load  (60  and  96  tracks),  for  threat  level 
two,  and  for  the  five -site  weapon  configuration. 

The  following  conditions  define  the  idealized  human  solution. 
Two  missile  assignment  modes  were  employed:  (1)  missiles  used 
early  on  priority  targets,  versus  (2)  missiles  used  only  as  back-up 
weapons.  Two  action  rates,  three  per  minute  versus  five  per  minute 
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were  employed  over  both  weapon  conditions  to  simulate  realistic  human 
assignment  capacities.  Aircraft  were  launched  either  two  on  one  at-a- 
trail  or  three  on  one  at-a-trail.  If  the  first  fighter  made  the  kill,  then 
the  second  was  permitted  to  continue  on  to  another  assignment.  This 
avoided  the  rule  of  lost -fighter  probabilities  being  assigned  until  the 
second  intercept  attempt  (cf  Appendix  I).  In  addition,  fighters  not 
"used  up"  in  trail  assignments  were  not  returned  to  base  for  rearming, 
but  could  be  diverted  to  another  assignment.  (The  commander  also  had 
these  options  in  the  experimental  runs).  In  every  case,  assignments 
were  made  first  against  bombers,  then  against  fighter -bombers,  then 
fighters,  and  then  other  inputs. 

The  "idealized  machine"  solution  approach  was  the  same  as  for 
"idealized  man"  except  that  zero  action  delay  was  assumed.  The 
"random  machine"  or  poor  machine  performance  solution  employed 
the  zero  time  delay,  but  tracks  were  assigned  on  in  order  of  appearance 
and  the  actions  (weapon  combinations)  were  assigned  to  them  at 
random. 

The  results  of  these  various  solutions  are  summarized,  below, 
in  Table  9.  As  might  have  been  expected,  the  zero  time  delay,  "idea¬ 
lized  machine"  approach  was  superior  in  terms  of  average  percentage 
of  kills  (93.  2  per  cent).  However,  kills  by  themselves  do  not  adequate¬ 
ly  describe  effectiveness  of  performance;  in  every  idealize dmachine 
solution  penetrations  occurred,  and  damage  was  assessed  in  the 
majority  of  cases.  The  obtained  weapon  per  kill  ratio  of  2.  8  to  1  is  of 
special  note  because  of  its  similarity  to  the  ftgure  given  by  the 
research  data.  We  have  already  observed  that  the  commanders' 
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TABU  9 


ANALYTICAL  SOLUTION*  TO  A  AMPLE  PAOBLIM* 


KILL#  WgAFOIM  DafD  DAMAGE  WEAPONS 

No.  of  %  TOTDT 

Trtck*  gUU  Total  B  TB  T  O  t  U  Total  tool  P  T  U  Kill 
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terminated  the  problem  series  at  96  tracks  with  just  over  a  3  to  1 
assignment  ratio,  and  with  the  downward  trend  still  evident. 

When  missiles  were  used  only  as  back-up,  idealized  human 
solutions  were  nearly  as  good  (90.  5  per  cent  kills)  as  the  idealized 
machine.  Alternatively,  when  missiles  were  used  early  in  the  track 
life  of  priority  targets,  the  overall  kill  percentage  dropped  to  87.7 
per  cent.  This  lower  kill  rate  may  be  accounted  for  in  part  by  the 
fact  that  an  extreme  range  missile  intercept  automatically  incurred 
the  application  of  one  of  the  kill  probability  downgrading  rules  (down¬ 
grading  of  kill  probability  due  to  extreme  range  assignment).  This 
could  have  a  marked  effect  if  it  occurred  across  most  of  the  missile 
assignments.  The  overall  achieved  weapons  per  kill  ratio  for  this 
machine  series  (missiles  early  on  priority  tracks)  was  almost  identical 
with  the  experimental  data  at  the  final  load  level. 

The  "random"  machine  solution  yielded  the  poorest  perform¬ 
ance.  Again  the  kill  figure  (55.  5  per  cent)  is  misleading  in  terms  of 
the  task  to  be  accomplished.  In  addition  to  the  low  proportion  of  kills, 
in  these  solutions  the  number  of  penetrations  and  amount  of  damage 
ran  quite  high.  The  weapons  per  kill  ratio  of  4.8  to  1,  here,  is  the 
highest  for  any  solution  and  is  a  further  indication  that  "poor"  decision 
performance  was  achieved  by  these  random  solutions. 

When  total  costs  are  considered  for  all  analytical  solutions, 
it  appears  that  superior  performance  was  gained  only  at  the  expense 
of  the  second  part  of  the  task  which  the  commander  had  to  -consider 
--that  of  maintaining  a  weapon  posture  which  could  counter  additional 
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attacks. 


One  other  feature  needs  to  be  pointed  out  here  also,  and  that  is 
the  risk  in  using  an  analytic  approach  such  as  this  without  considera¬ 
tion  of  the  replication  problem.  While  these  machine.^-  replication 
runs  compare  well  in  general,  several  extremes  in  performance  may 
be  found,  even  using  the  same  set  of  rules,  such  that  the  obtained 
ranges  of  the  order  of  24  percentages  points  were  obtained. 

Consideration  of  the  experimental  commanders'  performance  on 
the  problems  sampled  analyticaly  is  presented,  below,  in  Table  10 
and  indicates  that  they  turn  in  a  creditable  performance,  especially 
when  these  results  are  considered  in  terms  of  the  real-time  charac¬ 
ter  of  the  problems.  The  average  number  of  penetrations  for  the  96 
track  problem  is  the  same  for  the  analytic  "human"  solution  as  was 
obtained  in  simulated  battles.  Actual  executive  performance,  hrwever, 
showed  a  definite  saving  in  weapon  use  (188  vs.  227  for  "analytic 
humans").  The  greatest  number  of  weapons  used  in  the  "good"  solu¬ 
tions  did  not  reduce  materially  the  damage  score  sustained  in  the 
analytic  runs  as  compared  to  the  actual  human  performance. 

II.  ASSESSMENT  OF  EFFECTS  OF  INTERNAL  PROBLEM 
CONDITIONS 

One  goal  of  this  research  was  to  develop  understanding  of  the 
decision  process.  It  seems  likely  that  such  understanding,  if  derived 
from  the  study  of  complex  executive  functions  in  reasonable  simula¬ 
tion  of  actual  battle  constraints,  might  be  particularly  valuable  to 
the  designers  of  future  aerospace  threat  evaluation  and  action 
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selection  systems.  Today,  there  is  a  vast  growing  literature  on  two- 
choice  decisions  without  sequential  dependence  (30).  From  all  this 
experimental  and  theoretical  work,  alternative  models  of  human  pro¬ 
cessing  of  probabilistic  information  have  been  constructed  (2,  3). 

But,  there  is  no  way  to  apply  these  alternative  models  to  the  very 
complex  executive  function  in  surveillance.  Before  such  applications 
a  bridge  between  theory  and  the  particular  empirical  domain  must  be 
constructed.  It  is  suggested  that  research  of  the  present  sort  repre¬ 
sents  a  start  toward  building  this  needed  bridge.  This  work  aimed 
to  quantify  and  analyze  aspects  of  actual  executive  decision-making  with 
the  hope  that  the  resulting  systematic  description  of  behavior  would 
make  possible  the  application  and  test  of  particular  models  or  hypothe¬ 
tical  laws. 

To  initiate  the  analysis  of  the  effects  of  the  problem  variables, 
four  subject  commanders  were  sampled  in  terms  of  performance  on 
selected  tracks.  This  selection  was  designed  to  provide  a  sample  of 
damage  potential  vs.  non-damage  potential  tracks,  and  early  entry 
into  the  problems  when  load  was  low  and  weapon  inventory  was  high, 
versus  when  load  was  heavy  and  inventory  partly  depleted  and  some¬ 
what  variable. 

1.  Appropriateness  of  initial  action  selection  against  threat 
Table  11  summarizes  the  initial  assignments  of  the  sample 
of  commanders  by  threat  class,  damage  potential,  and  time  of  track 
entry  into  the  problems. 


-90- 


Distribution  of  First  Actions  for  Sample  of  Commande r s  b 


T3 

trJ 

o 

J 

08 

V 

a 

>. 

H 

«) 

H 

>S 


in 

CO 


CO 

eg 

in 


rg 


a>  co 

*  *5.2 

■*j  m  ra  Xl 
0  rt  u 

HWS< 


co  rg  o 


f  - 

Im  o  m  ^ 


*2  § 

«-jj 

0  nj  .3  u 
HW<< 

o 

CO 


O' 

n- 


co  o 

eg  in  r- 


eg  O' 


Tf  O  ^ 
in  f-H 
N 


<D 

aj  0>  «o 
•2  ^  co 
0  to 

H 

CO 


fi 
0 
•  H 
•M 

U 

< 


111 
^  I  *-•  O'  I©  CO  o4 
co  rg  co  co  ^ 


co 

sO  ^*w 


0 


rg  eg 


0> 


© 
© 
CO 

TO  4-» 

J  *3  s 

jS  «.s 

o.ii  tJ 


rg 

oo 

O' 


—I’M  T 

JSJ.S 

TT  H  4J 

0»H  (J 

e-fa  < 


rH 

oo 

1 

in 

7 

O 

HCO  |  — 1 

m  rg  1 

cr 

• 

rg  co  rg  i-H 

r-  r- 1 

r- 

cr 

CO 

* 

CO 

i 

i 

o 

^  co  j  o  00  00 


SO 

O' 

ffl 

tp 

CQ 


o 
^  eg 

m  co 


©  eg 

HHco  rg  eg  — i  so 

eg  —i 

^  ^  *-*  co  rg 


oo  O' 


oo  oo  r-  m  ^ 


o  co  rg 

oo  0s 


^  co  rg 
oo  *-i  — i  m 


nO  O' 


VO  00  oo 


© 


r  ^  O'  n 
M  00  00  J 


• 

sO 

vO 

rg 


O'  co 
i*om 

^  *-H  *-H 


oo 


O 

fa 

oo  CQ 

fa 


CQ  — 1 


co  rg 

o  m  in 

CO 

I'M  •— 

rg  — 

rj  <-i  M 

rH 

CO 

rg 

rg  oo 

co  r- 
^  >o  sO 

(M 

J° 

f-H  -sO 

f-H 

r-  ro  so 

f-H 

rj 

F-H  CO 

vO  O' 

co  O'  1 
so  r^-  r- 

Joo 

t 

TO 

^  r- 

GO  •— < 

f—  vO 

f-H 

r-t  in 

© 

^H  sO 

r-  r- 

•-h  oo  r- 

©  m  m 

*-h 

1  rH 

vO 

CM  fa 
r~ 

PQ 

fa 

CQ 

o 

fa 

°  CQ 
fa 

CQ  cm 


m  oo 
rg  oo  co 

T*  T$< 


in  —i 


co  rg  *-h  co  rg  h 

co 

D 


CQ 

CO 

•H 

2  * 
^  rH 

P 

flj  * 

mW 

TO 

6 

c3 


TO 

P 

U 

P 


—i  f>- 


c0  *--« 


00  ■<*  —1 


r^ 
—i  rg 


O'  O'  co 


^  ^  oo 

O' 
no  rg 


co  ^  no  rg 

co  rg  — i  po  rg  »-h 
(0 
a) 

•-H 

■H 

CQ 

CQ 


I 

c 

2 


p 

0)  TO 

S°W 

£ 

Q 


TO 

P 

U 

P 


rg  co 


co 

h-  r-  i-H  oo  — •  o 


in  - 
co  < 


in  co  co  m  m 
co  rg  — i  co  rg  —■ 


<u 

r-H 

•iH 

CQ 


SdJ 

TO 

£ 

TO 

Q 


TO 

P 

u 

p 


^  in 


co 

<d 


CO 

CQ 


i  <u  * 

cjooJ 

it 

(3 


TO 

P 

U 

P 


S< 

r  H  rH 

iS  s 

O  0 

H  H 


^  rH 

TO 

4-> 

o 

H 


rg  rg 

rH  in  vO 

O'  O' 

©  CO  CO 

11  5 

1  6 

28  3 
70  52 
98  55 

_-2.  8 

m  co 

»-•  rg 

1  —*  f-H  f-H 

rg 

'  so  r- 

1  •“»  C0  CO 

oo  rfi 

rg  so 
in  in 

00  rH 

rH  rg  co 
rg  in  r^- 

rg  rH 

cr  j_x 

O 

1-H  TO 

■/  p 

Kfi 

vO 


Generally,  a  heavy  assignment  was  made  against  bomber 
threats;  on  first  action,  as  is  shown  by  the  column  sum  of  Table  11. 
Again,  there  is  no  indication  that  learning  to  discriminate  the  "real" 
threat  has  taken  place.  Hence,  the  commander  was  not  able  to  select 
out  damage  potential  tracks  and  concentrate  on  them,  either  early  in 
each  problem  or,  for  that  matter,  later  on. 

The  typical  commanders'  strategy  usually  was  modified  for 
late  entry  bomber  tracks  in  that  missiles  were  employed  with  about 
the  same  frequency  as  aircraft  interceptors. 

Bomber  threats  increased  from  22  to  42  over  load,  while  num¬ 
ber  of  actions  (not  number  of  weapons  assigned)  against  bombers,  for 
the  sample  of  commanders,  increased  fropn  73  to  153.  Weapons 
committed  were  usually  at  the  rate  of  two  or  three  on  one.  Less  than 

20  per  cent  of  the  actions  are  on  a  one-toj-one 

/ 

threats. 

Fighter  bombers  were  handled  by  :he  commanders  in  a  much 
different  manner  than  bombers.  A  two -oi. -one  aircraft  attack  was  the 
most  frequent  action  regardless  of  damage  potential  or  whether  the 
track  entered  early  or  late  in  the  problem.  It  may  also  be  noted  tha^ 
on  initial  assignments  the  most  frequently  used  Interceptor  was  the 
Red  class  weapon.  This  can  match  the  speed  and  generally  the  altitude 
of  the  fighter -bomber  threats.  As  previously  stated,  the  input  speed 
apparently  determines  the  action  selected  against  the  class  of  threat, 

I 

Missiles  are  very  seldom  committed  against  the  fighter -bomber 


weapon  basis  on  bomber 
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threat,  although  in  this  study,  the  fighter -bomber  damage  potential 
was  one-half  that  of  the  bombers  and  hence,  very  important.  One- 
on-one  assignments  against  fighter-bombers  were  on  the  order  of  22 
per  cent  of  the  sample. 

Fighter  threat  assignments  were,  in  general,  the  same  as  for 
the  fighter-bombers.  Fighter  inputs  increased  from  14  to  24  across 
load  while  actions  committed  against  fighters  increased  from  37  to 
87;  more  than  a  two-fold  increase.  This  increase  is  general  across 
load,  however,  in  that  the  initial  rate  of  actions  to  tracks  sampled  is 
2.  8  to  1  and  this  rises  to  3.  3  to  1  for  the  final  load  level.  This  is 
further  evidence  in  support  of  the  concept  that  "sheer  number  of 
tracks"  pacesthe  subject  commanders. 

2.  Reassignment  against  threats  by  class 

Out  of  979  first  actions,  294  had  to  be  recommitted  for  a 
second  intercept,  and  81  required  a  third  action.  Negative  outcomes 
fed-back  to  the  commander  in  this  large  volume  may  pose 'special 
problems.  Threat  would  appear  to  go  up  considerably,  while  available 
time  to  process  would  be  considerably  lessened,  and  available  inven¬ 
tory  would  become  progressively  less  adequate. 

Reassignments  also  became  necessary  when  load  within  a 
problem  was  high.  This  increased  press  of  the  situation  might  be 
expected  to  alter  the  decision  making  capability  of  the  commanders. 
What  approach  or  strategy  does  he  then  employ  to  cope  with  the 
situation?  Table  12,  below,  summarizes  the  actions  that  were  taken 
on  second  and  third  assignment  against  the  sample  tracks. 
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For  second  assignment  against  bombers,  there  was  a  strong 
shift  to  two-on-one  and  three-on-one  missile  assignments.  Second 
assignments  against  fighter -bomber 8,  also,  exhibits  a  shift  to 
missiles,  but  usually  at  only  the  two -on -one  assignment  rate.  Second 
assignment  against  fighters  also  shows  the  shift  to  missiles,  across 
the  board.  This  rate  of  missile  assignment  increased  as  the  load 
increased.  Fighter  assignments  against  fighters  tended  to  drop  to  a 
one-on-one  approach  as  load  increased.  Second  assignments  on  other 
threat  tracks  at  high  load  have  essentially  dropped  out  occuring  in  only 
one  case.  Missiles  to  aircraft  committed  were  about  equally  divided 
for  second  assignments,  and  this  was  also  the  case  for  third  assign¬ 
ments. 

Third  assignments  on  bombers  showed  either  a  heavy  missile 
assignment  ora  low  (one-on-one)  aircraft  assignment.  In  the  latter 
case  the  bomber  probably  exhibited  little  threat  to  the  sites.  Anyway, 
it  has  been  shown  previously  that  considerable  capability  remained 
at  the  end  of  each  mission  and  could  have  been  used  against  this  class 
of  threat  if  the  commander  had  noted  danger  to  a  weapon  site.  Third 
assignments  against  fighter -bombers  followed  the  pattern  employed 
on  the  bombers  except  that  the  missile  assignments  were  usually  two- 
on-one.  This  mode  increased  in  frequency  with  increasing  problem 
load.  Aircraft  committed  against  fighter-bombers  were  generally  in 
a  two-on-one  mode.  This  increased  commitment  against  the  fighter- 
bomber  input  indicates  that  the  commanders  were  aware  of  fighter- 
bomber  threat,  but  attempted  to  counter  it  as  cheaply  as  possible 
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on  the  first  assignment,  while  considerable  time  remained  to  act  again, 
if  necessary. 

Incoming  fighter  tracks  also  were  assigned  against  the  second 
time  with  a  heavy  missile  commitment.  This  may  indicate  that  threat 
type  was  not  discriminated  well  by  the  commanders,  or  possibly  it 
may  mean  that  this  response  is  partly  an  experimental  artifact  due  to 
the  initial  appearance  of  fighters  near  the  weapon  sites.  The  fact  that 
heavy  aircraft  assignment  against  this  class  diminished  on  second 
assignment  may  indicate  that  some  evaluation  of  threat  level  has  taken 
place. 

The  third  assignment  on  fighters  also  showed  a  relatively  high 
commitment  ratefor  missiles.  When  used,  however,  most  of  the 
fighter  assignments  were  of  a  one-on-one  mode  for  the  third  assign¬ 
ment.  In  summary,  then,  it  can  be  seen  that  missiles  and  aircraft 
are  used  with  about  equal  frequency  for  both  the  second  and  third 
assignments. 

3.  Kill  probability  sampling  for  damage  vs  non-damage 

a*id  aaxiy  ys  late  entry  tracks 

In  Part  I,  results  for  kill  probability  sampling  were  shown 
to  remain  high  across  the  increasing  load,  in  general.  In  this  section, 
performance  against  a  sample  of  early  and  late  entry  tracks  which 
could  vs.  those  which  could  not  cause  damage  is  sampled  for  kill 
probability  achieved.  Table  13  summarizes  these  results  for  the  sam¬ 
ple  of  the  commanders. 

Bombers  in  general  had  the  higher  kill  probabilities  assigned, 
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rage  kill  probability  selected  for  sample  of  Commande 
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and  thia  is  consistent  across  the  increasing  problem  loads.  Within 
each  problem  the  difference  between  early  and  late  entry,  and  damage 
vs  non -damage  bombers,  is  probably  not  large  or  consistent  enough 
to  be  meaningful. 

The  average  fighter -bomber  kill  probability  that  was  selected 
tends  to  be  somewhat  lower  than  for  bombers.  This  holds  almost 
constant  across  loads.  Within  a  load  level,  the  late  entry  fighter - 
bomber  showed  a  slighly  lower  kill  probability  selected,  although, 
of  course,  with  this  small  sample  any  slight  differences  might  be  due 
to  chance. 

The  average  fighter  kill  probability  selected  holds  constant 
across  load  level.  There  are  no  apparent  differences  for  kill  prob¬ 
abilities  selected  against  the  fighter  inputs  for  early  vs  late  entry  or 
for  damage  vs  non -damage  tracks. 

Actions  recorded  against  the  remaining  "no  damage"  tracks 
show  that  attempts  were  made  to  kill  them  across  all  levels  of  load. 
The  actual  number  of  cases  is  very  small,  here,  but  nonetheless, 
these  actions  against  hostile,  but  harmless  tracks  raise  a  question 
of  the  level  of  evaluation  of  threat  achieved  by  the  commander.  It 
should  be  recognized,  however,  that  the  task  instructions  did  include 
the  injunction  to  seek  to  achieve  kills  on  hostile s  regardless  of  threat 
status,  so  the  executives  were  within  the  ground  rules. 

These  data  on  damage  versus  no  damage  tracks  show  that  the 
commanders  did  achieve  the  correct  ordering  of  kill  probabilities 
for  the  various  broad  classes  of  threat.  Actions  against  bombers 
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yield  a  mean  probability  of  .57,  against  fighter-bombers  the  figure 
is  down  to  .  50,  for  fighters  it  is  .47,  and  for  the  other  tracks  it  is  .  38. 

4.  Damage  assessed  against  a  sample  of  commanders 
An  examination  of  total  damage  assessed  against  each 

member  of  the  commander  sample  may  be  made  to  determine  whether 
they  achieved  a  relatively  uniform  success  in  this  phase  of  the  task  or 
if  wide  individual  differences  were  present.  If  one  or  more  individuals 
developed  serious  deficiences  in  action  selection  technique,  one  would 
expect  the  effect  of  such  inferior  assignments  practices  to  show  up  in 
this  score  of  damage  sustained. 

Examination  of  Table  14,  below,  indicates  that  the  spread  in 
individual  differences  was  quite  narrow  considering  the  complexities  of 
the  problems  and  the  potential  damage  presented.  While  inter-subject 
differences  were  small,  threat  levels  were  marked,  totalling  about  a 
two -fold  increase  in  damage  assessed.  This  is  alBo  the  situation  for 
increase  in  track  load;  hence,  it  does  not  appear  that  total  damage  was 
an  insensitive  measure.  The  scores  suggest,  however,  that  the  comm¬ 
anders  were  able  to  process  the  site  selection  aspect  of  the  problem 
in  either  the  five  or  the  eight  site  mode  about  equally  well.  In  general, 
these  damage  assessment  results  suggest  that  the  selected  commanders 
all  attained  competence  in  minimizing  damage.  Wide  individual  diff¬ 
erences  are  not  apparent,  and  the  damage  totals  represent  only  a  small 
fraction  of  the  damage  potential  that  existed. 

5,  Implications  of  the  results  for  decision-mak^j^  viewed^as 
a  process 
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It  may  be  suggested  that  performance  of  the  decision¬ 
making  tasks  in  this  set  of  problems  involves  two  different  levels  of 
judgment:  (1)  it  is  necessary  to  make  an  appropriate  action  choice 
against  each  individual  track,  and  (2)  the  executive  has  to  evaluate 
the  resulting  set  of  momentary  judgments  in  the  context  of  the  develop¬ 
ing  problem.  This  second  level  of  judgment  may  be  thought  Of^s  an 
examination  of  the  success  of  the  strategy  being  employed  and  possible 
selection  of  some  altered  assignment  rules.  Obviously,  its  optimum 
performance  should  be  based  on  all  the  information  elements  available 
since  all  are  known  to  be  relevant  to  prediction  of  the  future  progress 
of  the  problem.  In  addition  to  the  information  presented  dynamically 
on  the  various  displays,  which  may  govern  the  individual  scramble 
assignment  to  a  large  degree,  the  strategist  should  consider  the  prob¬ 
lem  ground  rules,  the  advance  intelligence  information,  and  successes 
and  failure  recalled  from  earlier  problems. 

It  is  the  impression  and  belief  of  the  team  of  investigators  that 
the  commanders  did,  in  fact,  perform  this  second  strategy  evaluation 
and  modification  level  of  judgment.  Using  the  information  elements 
listed,  above,  the  commanders  appeared  to  have  modified  the  strategy 
and  approach  to  solving  problems,  particularly  with  respect  to  points 
such  as  the  initial  use  of  aircraft  weapons  (the  approach  that  was 
developed  in  many  of  the  training  runs  and  which  may  have  been  related 

I 

to  the  commanders'  prior  experience  in  air  defense  operations). 

Analysis  of  the  debriefing  data  suggests,  however,  that  decision¬ 
making  at  the  strategy  level  is  not  entirely  a  conscious  process. 
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Particularly,  this  impression  may  be  gained  from  the  lack  of  descrip¬ 
tive  accuracy  in  the  commanders'  verbal  reports  of  success.  Initially, 
the  reports  of  adequacy  of  performance  were  too  optimistic.  Comm¬ 
anders  believed  that  they  had  done  better  than  the  objective  measures 
of  performance  indicated.  They  also  overestimated  the  accuracy  and 
utility  of  the  briefing  and  intelligence  data,  which  was  available  to  them 
during  the  problem  runs  only  in  the  form  of  memories.  At  the  end 
of  the  experiment,  however,  commanders'  estimates  had  become 
better  descriptions  of  the  facts. 

What  now  can  be  said  about  the  nature  of  the  decision  process 
studied  in  these  problems?  Three  major  features  appear  to  character¬ 
ize  the  process:  (1)  dynamic  sorting  of  categories  was  continuously 
required  in  terms  of  threat,  counter  weapons  and  locations,  rule 
structure,  and  other  data  that  had  been  provided;  (2)  there  was  dyna¬ 
mic  or  on-going  weighting  of  alternatives  selected  in  terms  of  the 
overall  outcomes  desired;  and  (3)  there  was  partial  lack  of  awareness 
or  at  least  inability  to  verbalize  reliably,  how  these  processes 
occurred. 

From  the  two  kinds  of  analysis  of  results  presented  in  this 
section,  it  seems  that  the  commanders  handled  items  one  and  two 
quite  well  in  terms  of  sorting  for  threat,  which  was  elementary  in 
this  case.  This  is  indicated  by  satisfication  of  the  matching  criteria, 
selecting  of  armaments,  yielding  appropriate  kill  probabilities, 
assignment  of  priority  to  the  tracks,  and  failure  to  process  but  a  very 
small  percentage  of  the  total  number  of  tracks.  While  doing  the  above, 
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commanders  were  able  to  select  reasonably  short  intercept  times, 
and  to  use  single  intercept  weapons  (A/C)  advantageously,  so  that  they 
would  return  to  inventory  in  time  to  be  ready  for  second  intercepts. 
Also,  they  were  able  to  work  efficiently  in  spite  of  the  two  ground  rules 
which  might  have  reduced  kill  probabilities.  This  efficiency  extended 
in  the  maximum  load  levels  to  the  point  that  they  were  experienced 
enough  to  achieve  kill  probabilities  equal  to  or  better  than  those 
achieved  during  the  low  load  problems.  Another  procedure  the  comm¬ 
anders  exercised  more  and  more  was  the  use  of  missiles  when  load 
dictated  and  faster  outcome  feedback  was  necessary. 

The  relatively  slow  development  of  an  accurate  appreciation 
of  fairly  crucial  information  elements  in  these  problems  suggests  a 
general  point  believed  to  be  of  some  importance.  This  point  is  that 
when  you  aie  dealing  with  complex  decisions  such  as  those  of  the  mili¬ 
tary  executive,  a  very  great  deal  of  practice  and  experience  is  to  be 
desired.  Mcsr  classical  experiments  in  decision-making  have  used 
tasks  of  such  simplicity,  e.  g.  ,  "guess  which  light  will  come  on  next, 
the  left  or  the  right,  "  that  a  beginner  might  quickly  learn  to  do  as  well 
as  an  old  hand.  But  this  is  not  the  way  decision  tasks  of  importance 
are  encountered  in  the  real  world.  True,  we  have  to  decide  which 
shoe  to  put  or.  fir  st  each  morning,  but  a  very  simple  strategy  will  suit 
the  problem.  Crucial  human  roles  in  surveillance  systems  are  quite 
different.  They  call  for  exercise  of  a  higher  level  of  judgment.  Thus, 
the  principal  accomplishment  of  this  study  may  be  that  it  has  provided 
some  baseline  data  to  show  that  a  fully  trained,  highly  selected  human 
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executive  can  perforin  well,  judged  by  comparison  to  various  machine 
or  analytic  problem  solutions,  in  higher  order  decision  functions, 
despite  task  load  factors  of  very  considerable  weight.  From  the  van¬ 
tage  of  these  empirical  statements,  it  should  be  possible  to  probe  the 
the  effects  of  a  number  of  procedures  and  experimental  factors  believed 
to  be  relevant  to  complex  decision  processes.  Additional  studies  in 
the  present  program  have  been  designed  with  that  goal.  Particularly, 
the  effects  of  error  and  system  noise,  in  general,  are  thought  to  be 
worthy  of  study,  for  the  reason  that  some  degree  of  unreliability  of 
surveillance  data  always  obtains,  and  costs  go  up  sharply  when  noise 
reductions  are  demanded.  To  make  design  tradeoffs  on  a  rational 
basis,  it  is  necessary  to  develop  a  decision  technology  that  takes 
account  of  the  implications  for  decision  effectiveness  of  the  various 
categories  of  input  and  processing  errors.  Those  studies  remain  to 
be  performed. 
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APPENDIX  1 


PROCEDURE: 

The  general  task  of  the  commander  or  decision  maker  in  this 
experiment  was  fourfold  as  follows: 

(1)  To  minimize  the  damage  to  the  weapon  areas 

(2)  To  destroy  a  maximum  number  of  threatening  vehicles 

(3)  To  conserve  counter  weapons  so  as  to  consume  a 
minimum  of  forces  consistent  with  objectives  (1)  and 
(2)  above 

(4)  To  develop  his  own  strategy  under  constraints  imposed 
by  the  ground  rules. 

Two  basic  certainties  existed  in  this  study;  the  commander 
knew  that  he  could  depend  on  (1)  highly  reliable  surveillance  data  and 
(2)  a  reliable  assessment  of  the  action  outcomes.  Two  basic  uncer¬ 
tainties  also  existed,  in  that,  (1)  the  threat  intelligence  data  was  pre¬ 
sented  over  a  range  of  probable  values,  and  (2)  there  were  variable 
kill  probability  outcome  distributions  for  the  weapons.  To  a  lesser 
extent,  uncertainty  existed  in  the  selection  of  the  variable  outcome 
times.  The  commander  knew  the  range  of  outcome  times  possible, 
and  as  a  sub-task,  he  had  to  select  weapons  which  would  yield  an  over¬ 
all  minimum  intercept  time. 

A.  Threat  Vehicle  Types 

In  the  present  experiment  there  were  four  broad  cate¬ 
gories  of  threatening  vehicles,  all  of  a  conventional  air-breathing  sort. 
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There  were: 


1.  ^ggjbe^JThreat^  -  There  were  three  categories  within 
this  class: 

(a)  High  Bombers  -  This  group  includes  bombers  of  high 
mission  speeds  (800  knots)  as  well  as  high  altitude  (65,  000  feet).  These 
might  be  countered  only  with  interceptor  missiles  or  "Green"  class 
fighters. 

(b)  Medium  Bombers  -  This  group  of  vehicles  includes 
bombers  whose  altitude  limit  is  lower  (55,  000  feet)  and  whose  mission 
profile  speed  is  700  knots.  The  appropriate  classof  manned  intercep¬ 
tor  is  the  Blue  type.  Green  class  fighters  may  be  committed  against 
this  class  input,  but  they  would  be  normally  considered  an  overmatch. 
Again,  all  ground-to-air  missiles  may  be  used. 

(c)  Low  Bombers  -  These  vehicles  have  an  altitude 
capability  of  45,  000  feet  and  speeds  of  the  order  of  600  knots.  The 
matching  class  fighter  is  the  Blue  type,  but  the  Red  infrared  seeker 
missile  armed  fighter  could  also  be  used.  Ground-to-air  missiles 
also  might  be  used. 

2.  Fighter  Bombers  -  There  was  one  class  of  fighter -bom¬ 
bers:  speed  was  500  knots  and  altitude  ran  to  45,000  feet.  Ground-to- 
air  missiles  could  be  used  against  this  class  threat  and  all  fighters 
with  the  one  exception  of  Red  machine -cannon  armed  class  (these  are 
restricted  to  40,  000  feet  altitude). 

3.  Fighte rs  -  Threatening  fighters  were  of  three  classes: 

(a)  High-  -65,  000  feet,  800  knots 
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(b)  Medium- -55,  000  feet,  700  knots 

(c)  Low--25-50,  000  feet,  500-600  knots 


4.  Other  threat-labeled  tracks  -  There  were  two  classes  of 
additional  tracks  in  this  category: 

(a)  Reconnaissance  'vehicles  -  65,  000  feet,  900  knots 

(b)  Troop  and  cargo  transports  -  to  25,  000  feet  and 
300  knots. 

B.  Counter  Weapon  Types  and  Sites 

The  counter  weapons  available  to  the  commander  were  of 
broad  types:  missiles  and  interceptor  aircraft.  These  were  located  in 
either  a  five-site  or  an  eight-site  distribution.  The  forward,  or  enemy 
ward,  were  designated  airfields  one  and  two.  Airfield  two  was  deleted 
in  the  five-site  mode.  Red  class  fighers  (of  somewhat  restricted  cap¬ 
ability')  were  the  only  counter  weapons  that  were  deployed  in  these 
locations.  The  intermediate  sites  some  distance  directly  behind  sites 
one  and  two  were  designated  as  airfields  three,  four  and  five.  Airfield 
five  was  deleted  in  the  five -site  mode.  The  rearmost  areas  were 
designated  missile  areas  A,  B  and  C.  Missile  site  C  was  deleted  in 
the  five-site  mode.  In  the  initial  instructions  it  was  pointed  out  to  the 
commanders  that  while  the  five -site  configurations  may  appear  the 
easier  of  the  two  modes,  the  threat  from  a  penetration  has  now  nearly 
doubled  and,  hence,  it  may  be  much  more  difficult  to  insure  survival 
of  his  weapon  capability. 

The  two  tapes  (containing  the  digital  inputs)  provided  two  diff- 

i 

erent  penetration  directions  on  per  tape.  The  commander  was  briefed! 
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in  terms  of  the  expected  strike  routes  for  the  main  hostile  thrust.  This 
was  from  right  to  left  across  his  display  for  tape  I.  and  from  upper 
left  to  lower  right  for  tape  II  --  the  more  complex  tape  in  terms  of 
threat  complexity  involved.  (Site  orientation  toward  "North"  or  upward 
on  the  display,  was  not  under  consideration  in  this  experiment). 

1.  Ground-to -Air -Missile 8  -  These  weapons  were  of  relative¬ 
ly  low  yield,  requring  proximity  contact.  They  were  employed  in  two 
modes: 

(a)  Long  range  --In  this  mode,  the  missiles  could 
reach  a  target  anywhere  in  the  situation  domain.  Intercept  times 
ranged  from  one  to  seven  minutes.  One,  two,  or  three  missiles  could 
be  launched  per  salvo. 

(b)  Short  range  --  The  range  in  this  mode  was  restricted 
to  targets  penetrating  a  fifty-mile  diameter  circle  surrounding  the  site. 
If  an  action  order  was  pending,  the  missiles  were  launched  at  penetra¬ 
tion  time.  Intercept  feedback  time  was  one  minute.  A  maximum  of 
three  salvos  of  tlree  missiles  each  could  be  launched  against  a  target. 
The  assumed  "missile  command"  had  the  same  digital  track  informa¬ 
tion,  but  had  no  autonomous  control  and  was  to  operate  only  under  the 
direction  of  the  area  command  in  these  exercises.  To  this  end.  de¬ 
commit  queries  were  sent  from  the  missile  sites,  when  in  the  short 
range  mode,  to  the  commander  when  a  track  under  this  assignment 
mode  had  faded. 

Only  three  automatic  missile  operating  conditions  were 

possible: 
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(1)  Missiles  assigned  to  low-priority  targets  would 


"hold  fire",  if  a  prior  aircraft  assignment  was  being  pursued  into  the 
missile  site  until  outcome  of  this  previous  assignment  became  known. 
This  rule  was  voided  when  a  priority  target  also  under  assignment 
penetrated  the  missile  site. 

(2)  A  long  range  guidance  mode  would  convert  and  fire, 
as  in  short  range  mode,  when  its  assigned  track  penetrated. 

(3)  A  three  salvo  sequence  automatically  fired  until  a 
"kill"  is  registered  or  the  sequence  is  completed. 

2.  Interceptor  Aircraft 

(a)  Green  class  fighter  (65,  000  feet)  --  This  was  a 
special  high  performance  interceptor.  It  operated  at  800  knots  aver¬ 
age  profile  spee$l  and  could  climb  to  65,000  feet  altitude  at  approxi¬ 
mately  40,000  feet  per  minute.  It  was  restricted  to  one  intercept 
before  returning  to  base  for  more  fuel  and  weapons.  However,  after 
a  15-minute  delay  a  previously  used  Green  class  fighter  was  available 
on  ready  status  for  scramble  again.  These  inerceptors  carried  one 
of  three  armament  systems:  machine  cannon,  infra-red  seeker 
missiles,  or  an  all-weather  system.  Green  was  the  matching  class 
for  high  performance  threats  and  the  fighers  were  found  at  airfields 
four  and  five  only. 

(b)  Blue  class  aircraft  (55,  000  feet  and  700  knots)  -- 
This  was  a  general  purpose  vehicle  which  could  remain  airborne  for 
the  life  of  the  problem  and  could  run  repeated  intercepts  on  the  basis 
of  ample  armament  and  fuel  stores.  Thus,  this  weapon  could  be 
diverted  from  an  airborne  station  immediately  after  an  intercept  attack. 
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The  three  armament  systems  listed  for  Green  also  apply  here.  These 
aircraft  were  based  at  airfield  three,  four  and  five. 

(c)  Red  class  aircraft  (40,  000-45,  000  feet  and  600  knots) 
--These  were  small  interdiction  weapons  operating  out  of  two  forward 
area  dirt  strip  configurations.  They  were  released  to  the  commander 
for  any  necessary  defensive  operation.  They  could  fly  only  one  inter¬ 
cept  mission,  and  they  had  to  RTB  for  rearming  and  refueling.  They 
were  armed  with  either  a  machine  cannon  configuration  or  IR  rockets 
(the  service  ceiling  was  40,  000  feet  when  the  machine  cannon  mode 
was  employed). 

C.  Scoring  Procedures 

In  general,  where  "matching"  class  fighters  or  ground-to- 
air  missiles  are  assignecK-by  the  commander,  scoring  presents  no 
problems.  An  "undermatch"  (assignment  of  a  Red  class  weapon  to  a 
threat  for  which  the  matching  class  is  Green,  for  example)  is  scored 
as  a  miss.  Overmatching,  as  described  above  will  ordinarily  increase 
kill  probability  and  yield  a  slight  advantage  in  intercept  time.  The 
commander  was  not  penalized  for  overmatching  by  the  scoring  proce¬ 
dure,  but  ran  the  risk  of  using -up  these  weapons  on  lower  level  threats 
is  used  indiscriminately. 

Each  possible  action  outcome  had  to  be  established  and 
located  in  a  readily  accessable  form  for  the  referees  so  that  these 
referees  could  provide  to  the  commander  the  evaluation  resulting  in 
dynamic  feedback  of  action  outcomes  during  the  course  of  the  tactical 
problems. 
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D.  Weapon  Assignment  Modes  and  Procedures 


One  of  the  most  important  points  to  be  remembered  concerning 
weapon  assignments,  aside  from  selection  of  appropriate  weapons,  is 
the  standardization  of  the  assignment  message  format.  Standardiza¬ 
tion  in  this  respect  is  of  the  utmost  importance  from  the  experimental, 
as  well  as  from  the  practical,  operational  point  of  view.  Experimen¬ 
tally,  only  standard  messages  can  be  handled  with  the  required  speed 
and  accuracy  by  the  clerks  and  referees.  However,  convenient  a  non¬ 
standard  message  is  to  the  commander,  it  slows  down  or  confuses  the 
execution  of  the  assignment,  resulting  ultimately  in  penalty  to  the 
commander.  In  addition,  future  systems  will  require  communications 
with  machines  in  language  that  the  machines  can  understand.  Non¬ 
standard  messages  are  understood  and  executed  even  less  well  (i.e.  , 
not  at  all)  by  machines  than  by  men  (as  in  the  present  experiment). 
Therefore,  an  essential  part  of  the  procedure  is  the  use  of  rigid,  but 
efficient  message  formats.  The  assignment  message  consisted  of  the 
track  number  assigned  against,  the  weapon  kind  to  be  used,  the  site 
to  be  drawn  from,  its  armament  category,  the  number  to  be  used,  and 
an  indication  of  priority  of  the  assignment  on  a  four-point  scale  (e.g.  , 
Track  84,  Green  three,  two  IR's,  priority  one). 

Weapons  may  be  assigned  one,  two,  or  three  at  a  time  against 
individual  threats.  In  addition,  "trail"  type  assignments  of  two  or 
three  successive  aircraft  or  missiles  may  be  used.  If  the  first  air¬ 
craft  kills,  the  rest  go  to  airborne -available  status.  In  the  case  of  the 
missiles,  all  are  fired  and  detonated  regardless  of  an  initial  kill  by 
the  first  weapon.  Due  to  the  actual  vagaries  of  surveillance  data  and 
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close  control  performance  (i.  e.  ,  a  prior  failure  of  control  might  still 
add  data  to  improve  a  subsequent  attempt),  trail  type  assignments 
generally  result  in  higher  kill  probabilities,  overall. 

Kill  probabilities  for  Green  type  aircraft  varied  from  .  2  to  .  8, 
depending  on  number  assigned,  armament  type,  delay  in  assignment, 
or  distance  to  go.  Blue  class  was  .  2  to  .8,  and  Red  class  .  1  to  .8. 

The  more  weapons  assigned  (up  to  three,  either  together  or  in  "trail", ) 
the  higher  the  potency  of  the  armament,  and  the  sooner  after  onset  of 
the  threat  the  higher  the  probability  of  kill. 

The  commanders  were  given  the  following  operational  ground 
rules  to  govern  their  deployment  of  the  counter-weapons: 

1.  Scramble  on  all  hostile  tracks, 

2.  Scramble  on  unknowns  as  hostile s  after  the  unknowns  had 
been  carried  by  the  system  for  one  minute. 

3.  Apply  matching  criteria  by  assigning  a  weapon  combina¬ 
tion  that  can  make  the  target  altitude  and  is  reasonably  close  to  the 
target's  speed. 

4.  Indicate  the  priority  of  each  assignment  as  follows: 

Bomber  designated  as  1,  Fighter -bomber  as  2,  Fighter  as  3,  and 
other  categories  (cargo,  troop,  and  reconnaissance)  as  4,  except 
where  the  commander  wishes  to  indicate  the  downgrading  of  certain 
assignments  to  indicate  their  presenting  a  lesser  threat. 

5.  Weapon  a ssignment  permits  one,  two,  or  three  aircraft 
or  long  range  missiles  to  be  committed  on  any  one  track  per  assign¬ 
ment  (except  for  the  short  range  missile  mode--where  a  maximum  of 
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three  missiles  and  three  salvos  may  be  committed).  Unlimited 
successive  assignments  per  track  are  permitted,  but  the  commander 
must  issue  a  new  scramble  order  for  each  new  assignment. 

6.  Observe  progress  of  friendly  traffic.  If  a  scramble  is 
assigned  inadvertently,  the  commander  will  receive  a  decommit  query. 

7.  Missile  interceptors.  Intercept  time  for  the  long  range 
mode  range  fromone  to  seven  minutes.  "Target  going-out-of-range" 
signal  will  be  fed  back  in  four  minutes.  In  the  short  range  mode,  firing 
can  commmce  anytime  after  perimeter  penetration,  but  kill  probability 
will  be  reduced  by  .  1  if  the  assignment  occurs  over  five  minutes  after 
penetration. 

8.  Interceptor  aircraft.  Intercept  times  will  range  from 
three  to  twenty  minutes.  A  "too -late -to -make  intercept"  signal  will  be 
given  10  minutes  after  scramble  if  the  target  will  be  out  of  range  in  20 
minutes.  A  "tail-chase"  signal  will  also  be  given  in  10  minutes  (four 
minutes  in  some  special  cases)  if  the  intercept  will  result  in  an  im¬ 
possible  tail-chase  at  20  minutes. 

9.  Damage  assessment: 

(a)  Bomber  penetration  (defined  as  25  miles  from  site) 
will  cause  loss  of  one-half  of  the  weapons  at  that  site  at  timeof  pene¬ 
tration  (weapons  on  the  ground  only). 

(b)  Fighter -bomber  penetration  will  cause  the  loss  of 
one -fourth  of  the  ground-based  weapons  (penetration  radius  same  as 
above). 

(c)  Fighter  penetration  will  cause  the  loss  of  one  airborne 
weapon,  if  any.  In  some  cases  it  will  be  a  weapon  on  the  ground  that 
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is  lost,  or  in  some  cases  it  will  be  a  missile,  if  a  missile  site  is  hit. 


(d)  Whatever  penalty  is  appropriate  will  be  assigned 
at  three  minutes  after  penetration,  with  a  few  exceptions  where  a 
longer  delay  is  introduced.  The  commander  will  be  notified  by  a 
priority  message,  and  also  the  loss  will  be  posted  in  the  appropriate 
location  on  the  status  boards. 

(e)  Damage  can  occur  if  Initial  Point  is  reached  before 
kill  time  (although  post-drop  intercepts  are  possible. 

(f)  Hostile  drop  success  was  stated  at..  8  (but  was  not 

applied). 

(g)  Site  evacuation  procedures: 

1.  Evacuation  to  CAP  will  not  be  permitted  until 
site  penetration  is  pending,  on  the  grounds  that  the  system  will  tend  to 
become  loaded  and  the  value  of  the  dispersal  at  the  fields  would  be 
lessened  thereby. 

2.  On  evacuation,  aircraft  will  be  flown  off  at  the 
rate  of  three  per  minute  with  normal  scrambles  still  permitted  until 
all  aircraft  are  launched. 

3.  In  the  case  of  short  mission  profile  weapons, 
they  will  orbit  15  minutes  and  then  return  to  base.  If  two  drops  have 
been  successful,  the  site  will  be  closed  out  and  the  weapons  will  RTB 
to  their  companion  site,  if  intact.  If  not,  they  will  be  staged  to  the 
nearest  rearward  area. 

(h)  System  performance  confidence  in  terms  of  detec¬ 
tion,  tracking,  identification,  and  communications  are  100%  at  the 
start  of  the  mission. 
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E.  Intelligence  Briefing 


The  commander  was  instructed  that  previous  intelligence 
and  surveillance  data  indicates  specific  threat-type  activity  at  several 
hostile  locations.  On  the  basis  of  these  and  other  previous  data,  an 
estimate  of  the  expected  hostile  commitment  could  be  made.  However, 
intelligence  might  not  be  aware  of  new  areas  of  activity,  so  only  gross 
estimates  were  quoted  to  him.  The  table  below  indicates  the  distribu¬ 
tion  of  estimated  threats,  confidence  limits  of  these  estimates,  and 
the  actual  threat  presented  for  each  mission.  The  commander  was 
briefed  on  only  the  estimated  threat  and  its  confidence  level,  in  terms 
of  expected  commitment  rates.  Also,  during  the  briefing,  the  comm¬ 
ander  was  given,  for  review  against  the  expected  threat  commitments, 
the  exact  distribution  of  weapons  available  for  use  in  the  mission, 

(See  additional  table  below). 

F.  Pre-mission  Training 

Each  commander  had  seven  runs  (through  a  96 -track  prob¬ 
lem)  with  a  two -site  weapon  problem.  The  missile  and  fighter-inter¬ 
ceptors  were  located  at  only  the  two  bases,  and  hence,  the  problems 
were  simple  in  that  only  two  bases  required  protection.  Feedback  on 
intercept  outcomes  was  provided  at  a  constant  time  after  assignment 
(six  minutes),  and  the  range  of  probabilistic  outcome  values  was  not 
extensive.  The  values  ranged  only  from  .45  to  .99  for  the  weapons 
under  the  commanders  control  or  .1  for  an  automatic  missile  fire  mode 
over  which  he  had  no  control.  The  latter  mode  yielded  some  additional 
kills  thereby  providing  the  commander  with  the  option  to  decommit 


-119- 


weapons  previously  assigned  as  no  longer  necessary  against  a  killed 
track  assignment. 

The  second  phase  of  training  consisted  in  presenting  a  more 
complex  level  of  problem.  Eight  weapon  sites  were  now  employed, 
consisting  of  five  aircraft  and  three  missile  sites.  The  automatic 
missile  firing  feature  was  dropped,  and  the  commander  now  had  to 
generate  all  weapon  assignments.  At  this  time  the  broader  (from 
0.  0  to  1.0)  distribution  of  probabilistic  outcomes  was  introduced  along 
with  variable  outcome  times  for  a  more  realistic  feedback  of  results 
of  the  commanders  action  selections.  This  tended  to  force  the  comm¬ 
ander  to  sharpen  his  tactics  to  preclude  choosing  assignments  with 
low  payoff.  The  commander  had  to  pay  much  closer  attention  to  all 
information  sources  provided.  The  commanders  actions  at  this  level 
of  problem  complexity  were  processed  by  the  introduction  of  the  pre¬ 
programmed  referee  sheets.  Some  debugging  had  to  be  done,  proce¬ 
dures  changed,  and  level  of  skill  developed  by  the  referees  to  service 
the  commander  as  he  progressed  from  loads  of  24  through  72  tracks. 
On  some  occasions,  when  the  experimenter  felt  some  commanders 
needed  more  practice  or  were  in  doubt  about  some  of  the  rules,  addi¬ 
tional  runs  were  required  to  bring  individuals  up  to  the  general  level 
of  performance. 
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30  to  70  per  cent  of  the  potential  threat,  and  that  the  range 
to  60  per  cent  of  the  figher  potential  threat. 


table  a 


Distribution  of  Damage  Potential  Tracks 
by  Load  and  Threat  Level 

TRACK  LOAD 


60 

72 

84 

96 

Hostile  Track  Load 

52 

61 

74 

85 

THREAT 

Threat 

LEVEL 

Type 

I 

Bomber 

4 

7 

6 

8 

Fighter- 

Bomber 

1 

0 

1 

2 

Fighter 

10 

3 

11 

12 

15 

10 

18 

22 

Hostile  Track  Load 

THREAT 

LEVEL 

Threat 

Type 

60 

72 

84 

96 

II 

Bomber 

4 

10 

12 

15 

F  ighter- 
Bomber 

0 

2 

0 

2 

Fighter 

11 

13 

17 

20 

15 

25 

29 

37 
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APPENDIX  II 


SIMULATION  FACILITY 

This  appendix  provides  a  functional  description  of  the  equipment 
utilized  to  prepare  and  prosecute  this  experiment,  details  of  the  track 
situation,  and  the  experimental  design  that  organized  the  data  collection. 

A.  Equipment 

1.  Analog  equipment  description: 

{a)  Moving  Radar  Target  Generator,  15- J1C( modified) 
The  moving  radar  target  generator  was  used  as  a 
new  video  information  source  for  the  simulated  situations  in  the  prob¬ 
lems.  The  signals  simulate  moving  aircraft  or  missiles,  the  speed 
ranges  being  dependent  on  the  scale  factors  used  in  the  system.  Ini¬ 
tial  starting  point,  speed,  course  direction,  and  rate  of  turn  are  in¬ 
serted  manually.  Antenna  rotationinformation  is  furnished  by  the 
antenna  rotation  simulator  to  these  devices  as  well  as  the  video  graph, 
AN/UPA-35,  and  the  Cartrac  Common.  Since  the  Cartracs  had  a 
display  coverage  of  150  miles  radius,  and  the  digital  display  gave  a 
300  by  300-mile  presentation,  many  tracks  had  to  be  converted  by 
off-setting  on  analog  to  digital  conversion  in  order  to  utilize  the  total 
display  area  of  the  digital  system, 

(b)  PPI  radar  scope  AN/UPA-35 

The  AN/UPA-3  5  is  a  monitor  scope  used  with  the 
15-JIC's  when  recording  tracks  on  a  magnetic  tape  to  prepare  a 
mission.  Ir.  this  process  the  monitor  scope  was  used  to  check  the 
azimuth,  range,  and  velocity  of  each  individual  track. 
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(c)  Radar  indicating  equipment  (Video  Graph) 

The  Video  Graph  (V.  G.  scope)  is  similar  to  a 

regular  PPI  scope  except  it  used  a  "dark  trace"  CRT  and  an  optical 
projection  system  for  displaying  the  images  on  a  horizontal  viewing 
screen.  The  surface  was  covered  with  tracing  paper  upon  which  the 
mission  tracks  were  plotted  in  real-time.  Thus,  a  hard  copy  record 
of  the  track  histories  became  available  for  analysis.  The  Video  Graph 
equipment  is  self-contained  requiring  only  video,  trigger,  and  antenna 
rotation  information. 

(d)  Cartrac  common 

The  cartrac  common  was  used  as  a  buffer  unit 
between  the  radar  simulation  equipment  and  the  cartrac  consoles.  It 
supplied  the  sawtooth  wave  voltages  needed  for  coordinate  conversion, 
provided  synchronization  between  the  antenna  rotation  simulator  and 
the  cartrac,  and  peaked  and  amplixied  the  video  information. 

(e)  Cartrac  console 

Each  cartrac  (cartesian  tracking)  consoles  con¬ 
sists  of  a  PPI  scope  plus  twelve  channels,  each  of  which  was  capable  of 
tracking  a  single  target.  The  electronic  gate  of  a  tracking  channel, 
(normally  3  miles  square)  was  assigned  to  a  target  manually  Dy  means 
of  a  "joystick".  First,  the  operator  positioned  a  spot  on  the  desired 
blip  by  moving  the  joystick,  and  then  he  assigned  a  particular  tracking 
channel  by  specific  motion  of  an  assignment  switch.  An  automatic 
correcting  circuit  in  the  assigned  cartrac  channel  then  kept  the  gate 
positioned  so  that  the  target  was  automatically  tracked.  The  output 
for  each  channel,  which  is  in  X^y  analog  form,  was  then  sera  to  the 
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coder  for  conversion  to  digital  form. 

(f)  Movement  identification  officers  (MIO)  console 

The  MIO  console  was  used  in  conjunction  with  the 
data  assignment  panel  to  indicate  the  status  of  the  twenty-four  tracking 
units.  Individual  cartrac  gates  may  be  identified  by  two  methods:  (1) 
depressing  a  numbered  button  corresponding  to  the  number  of  the 
tracking  channel,  and  (2)  by  placing  the  digital  spot  on  the  PPI  on  the 
gated  target  to  be  identified.  Numbered  lights  are  used  to  indicate  the 
status  of  a  cartrac  unit,  i.  e.  ,  hangared,  tracking  new  target,  or  track¬ 
ing  with  auxiliary  data  assigned. 

2.  Digital  Equipment  Description: 

(a)  Data  assignment  panel  (DAP) 

The  data  assignment  panel  provided  a  means  to 
insert,  manually,  an  identifying  numbe  r  (tag  number),  altitude,  speed, 
information,  target  type  (category  and  IFF  identification),  and  a  per¬ 
sonal  identification  number.  All  data  other  than  height  and  speed  were 
fed  to  the  coder  in  a  digital  format.  The  two  exceptions,  height  and 
speed,  were  set  by  potentiometers  and  converted  to  digital  information 
in  the  coder. 

(b)  ^ajjje-j‘ecioirde<r> 

The  tape  recorder  used  was  a  dual-channel.  Model 
350,  Ampex  Recorder.  One  charnel  contained  timing  pulses  used  for 
synchronizing  data  flow,  and  the  adjacent  channel  contained  problem 
data. 

(c )  One-hundred  target  tape  generator 

The  one -hundred  target  generator  was  developed 
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to  control  the  flow  of  data  between  the  recorder  and  the  rest  ol  fne 
system.  It  took  the  timing  pulses  from  the  recorder  and  sent  a  data 
transmission  signal  to  the  digital  communications  unit  to  record. 

This  was  necessary  because  targets  were  generated  in  blocks  of  12, 
and  the  problem  tapes  for  this  experiment  contained  9t>  targets.  A 
set  of  switches  determined  which  time  slot  on  the  tap 3  was  being  re¬ 
corded.  This  same  set  of  switches  allowed  the  experimenter  to  select 
numerous  combinations  of  target  loads  during  playback.  This  device 
may  be  considered  the  heart  of  the  system. 

(d)  Codei 

The  coder  was  essentially  an  analog -to -digital 
converter.  It  accepted  the  analog  gate  voltages  and  converted  them 
into  a  digital  message  format.  This  format  also  contained  auxiliary 
information  that  had  been  inserted  through  the  DAP.  A  target  message 
consisted  of  five  words  each  word  containing  ten  digits. 

(e)  Digital  communications  unit 

The  function  of  the  digital  communications  unit 
was  to  provide  timing  signals  to  the  coder  and  to  the  drum  storage 
unit.  These  sigrals  were  needed  for  the  formation  of  the  digital 
messages  in  the  coder  and  tc  control  the  record-playback  circuits  it- 
the  drum  storage  unit:. 

(f)  Drum  storage  uni1. 

The  purpose  of  'he  drum  storage  was  to  store  the 
position  and  auxiliary  da*a  in  a  manner  suitable  for  visual  presenta¬ 
tion.  Digital  to  analog  conversion  circuits  were  included  in  this  unit 
for  the  X-Y  position  voltages  needed  by  the  CRT  deflection  plates 
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and  there  were  matrices  for  binary  to  octal  conversion  of  the  auxiliary 
data. 

(g)  Digital  display  unit 

The  digital  display  consisted  of  a  cathode  ray  tube, 
with  a  long  persistence  phosphor,  and  in-line  display  for  the  display 
of  auxiliary  information.  The  system  cycling  time  was  sixteen  seconds. 
The  long  persistence  phosphor  left  a  "trail"  and  gave  a  visible  history 
of  the  tracks.  An  overlay  was  used  to  indicate  geographical  features 
and  military  targets,  such  as  bases  and  other  areas  to  be  defended. 

Auxiliary  data  was  called  up  for  display  by  aiming  a  light  gun 
at  a  particular  target  and  squeezing  the  trigger  button.  When  this  was 
done  tag  number,  velocity,  altitude,  category,  and  personal  identity 
appeared  on  the  in-line  display.  It  was  also  possible  to  display  all 
target  position  blips  of  one  specific  category,  i.  e.  ,  fighter,  fighter- 
bomber,  etc.  ,  or  one  particular  identification  class,  eliminating  from 
the  main  display  surface  all  other  blips. 

(h)  Digital  clock 

The  digital  clock  was  used  to  provide  a  source  of 
displayed  time  to  use  as  a  reference  base  throughout  the  mission.  The 
sixty-cycle  power  line  or  timing  pulses  stored  on  one  channel  of  the 
tape  could  also  be  used  as  a  timing  source.  Several  clocks  synchroni¬ 
zed  to  a  master  unit  were  used  to  insure  a  common  time  base  in 
refereeing  and  data  collection. 

3.  Ancillary  Equipment: 

(a)  Status  boards 

Plexiglas  tcte  boards  were  used  to  keep  a  running 


-127- 


account  of  the  weapons  inventory  during  the  missions.  They  consisted 
of  edgelit  boards  with  a  total  area  of  approximately  thirty  square  feet. 
Information  was  displayed  by  writing  on  the  backs  of  the  boards  with 
colored  grease  pencils,  (cf  diagram  at  end  of  this  Appendix). 

(b)  Communication  system 

The  telephone  system  consisted  of  a  power  supply, 
distribution  cabinet,  and  several  AN/GTA-6  telephone  boxes.  The 
arrangement  was  flexible,  capable  of  being  used  for  communications 
between  the  operator  and  the  refereeing  section  and  among  the  referees 
and  status  board  operators. 

B.  Experimental  Problem  Preparation 
1.  Track  Situation  Recordings 
(a)  Recording  procedures 

Tapes  were  generated,  in  units  of  twelve  targets  eight 
successive  times  to  give  a  total  of  ninety-six  targets  per  tape.  Position 
information,  in  polar  video  form,  was  generated  by  the  radar  simula¬ 
tors  and  sent  to  the  cartrac  common,  AN/UPA-35,  PPI  scope,  and  the 
Video  Graphic  scope.  The  cartrac  common  then  re -transmitted  this 
information  to  the  cartrac  and  MIO  consoles. 

Operators  at  the  two  cartrac  consoles  manually  assigned  tracking 
gates  to  ail  the  targets.  A  tag  number  and  auxiliary  information  for 
each  such  track  was  then  inserted  at  the  data  assignment  panel  by  an 
operator  who  used  the  MIO  console  to  call  up  the  identifying  number 
of  each  individual  track  and  gate  as  it  came  into  view  on  his  PPI. 

The  digital  coder  converted  the  X-Y  analog  position  voltages  of 
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each  tracking  gate  to  digital  form.  Height  and  speed  from  the  auxiliary 
data  panel  were  also  converted  to  digital  form  in  the  coder.  The  coder 
then  built  these  into  a  message  structure  for  each  target  and  sent  them 
to  the  digital  communication  unit. 

Timing  pulses  from  the  tape  recorder  were  sent  to  the  100-target 
generator  which  in  turn  sent  a  data  transmission  signal  as  previously 
discussed.  When  ordered,  the  digital  communication  unit  sampled 
the  coder  and  transmitted  the  message  to  the  tape  recorder,  where  the 
information  was  recorded  in  the  proper  block  of  the  tape. 

Two  basic  tape  recordings  of  track  situation  and  auxiliary  data 
were  prepared  for  the  experimental  runs  to  be  used  for  data  collection 
purpose.  A  third  tape  was  prepared  for  training  and  system  checkouts. 
Each  tape  contained  96  complete  tracks.  The  proportion  of  tracks  of 
different  types  will  be  found  listed  below.  Description  of  the  charac¬ 
teristics  of  different  classes  of  threatening  tracks  and  friendlies  may 
be  found  in  Appendix  I. 

During  track  generation,  the  targets  were  manually  plotted  on 
the  V. G.  indicator  to  monitor  how  well  the  simulator  crew  followed 
the  problem  scripts.  Since  many  errors  could  be  introduced  to  the 
data  stored  on  the  magnetic  tapes  from  mis-aiignment  of  the  simula¬ 
tors,  perturbations  in  the  cartrac  tracking,  errors  in  use  of  the 
data  assignment  panel  and  subsequent  noise  in  the  digital  system,  it 
was  necessary  to  check  all  aspects  of  these  stored  data. 
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Constitution  of  the  situation  tapes  by  numbers  of  tracks: 


Hostile 

Tape  I 

Tape  II 

Tape  III 

Training 

Data 

Data 

Bomber 

22 

31 

34 

F  ighte  r  -Bombe  r 

27 

20 

21 

Fighter 

23 

31 

38 

Other  tracks 
(recon.  ,  Troops, 

& Cargo) 

2 

3 

3 

F  riendly 

Bomber 

7 

4 

0 

Fighter -Bomber 

13 

4 

0 

Fighter 

2 

3 

0 

Other  tracks 
(  recon.  ,  T roops, 
Cargos) 

0 

0 

0 

96 

96 

96 

(b)  Data  check-out 

Three  methods  of  data  check-out  were  employed, 
in  addition  to  perusal  of  the  track  plots  on  the  V.G.  indicator  as 
follows: 

(1)  Variplotter  --  After  a  tape  had  been  generated, 
it  was  played  back  into  a  high  speed  read-out  and  graphical  recording 
instrument  at  an  eight -to -one  speed  ratia  This  was  done  for  all 
tracks  since  a  small  error  could  mean  the  difference  between  penetra¬ 
ting  and  not  penetrating  a  defined  target  area.  On  repeated  rerun  to 
sample  tracks  (both  tracks  that  followed  the  original  script  well  and 
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others  that  had  system  noise  introduced  during  recordings),  the  spread 
around  most  position  points  very  seldom  exceeded  two -and -one -half 
miles.  (This  small  error  can  be  accounted  for  in  the  mechanical 
back-lash  of  the  variplotter  itself.)  This  recording  also  furnished  a 
hard-copy  of  the  taped  track,  a  record  which  was  used  later  to  score 
all  possible  intercept  combinations. 

(2)  Computer  --  Since  a  computer  program  was 
available  from  another  effort  (24  ),  it  was  felt  desirable  to  play  these 
mission  tapes  into  the  computer  and  sample  the  stored  digital  informa¬ 
tion  independently  on  the  playback  elements  of  the  tactical  system. 

The  first  run  sampled  the  position  data  on  each  track  for  every  other 
(odd)  system  cycle.  The  second  run  sampled  all  the  stored  ancillary 
data.  Whenever  doubt  arose  about  any  of  the  track  data,  the  computer 
record  was  used  to  resolve  the  issue. 

(3)  Error  check  during  mission  --In  addition  to 
the  above  error  checks,  it  was  necessary  to  assess  system  perform¬ 
ance  during  playback  of  each  mission.  To  this  end  an  experimenter 
recorded  the  tag  and  ancillary  data  associated  with  each  action  that 
was  reported  and  also  for  most  of  the  track  interrogations.  This 
record  was  then  checked  against  the  preprogrammed  data.  Error 

was  calculated  to  be  less  than  or><*  pec-4-^nt.  -Alignment-  srrc-r  wa-s- - 

constantly  checked  by  an  experimenter  during  each  mission  to  insure 
that  site  penetration  occurred  as  planned. 

2.  Action  outcome  scoring: 

(a)  Computation  of  intercept  points  and  outcome  delay  times 
Intercept  points  were  computed  from  each  applicable 
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missile  and  aircraft  site  for  two  time  intervals  of  track  life  (action 
assigned  during  the  first  five  minutes  of  track  life  and  action  assigned 
after  five  minutes).  The  basis  for  this  computation  was  the  least  time 
intercept  based  on  the  position  data  from  the  variplotter.  Consideration 
was  given  to  climb-out  and  positioning  problems  associated  with  each 
armament  configuration.  The  gun-armed  fighter,  for  example,  had  to 
be  close  atrail  for  a  successful  quartering  stern  attack.  The  IR-seek- 
er,  missile  armed  aircraft  had  to  be  positioned  for  a  snap-up  tail-cone 
or  trail  tail-cone  attack,  but  could  be  much  further  astern  and  still 
permit  the  IR  missile  to  kill.  The  All-Weather  system  interceptor 
assumed  capability  to  attack  successfully  from  any  quarter,  given 
reasonable  positioning.  In  the  case  of  the  missile  (long  or  short  range 
mode)  any  attack  angle  was  possible  and  the  intercepts  were  so  scored. 
Intercept  time  for  aircraft  ranged  from  three  to  twenty -minutes  and 
from  one  to  seven-minutes  for  missiles. 

Tail  chases  intercepts  were  terminated  after  twenty -minutes.  A 
signal  was  sent  to  the  commander  ten-minutes  after  assignment  on 
any  intercept  that  was  detected  to  be  ending  in  an  impossible  tail  chase. 
This  "probable  miss"  signal  was  sent  at  ten  minutes,  and  was  also 
sent  on  actions  which  would  be  "out-of-range"  in  twenty  minutes.  The 
commander  then  could  decide  to  continue  engagement  or  he  could 
decide  to  decommit. 

(b)  Generation  of  weapon -kill  probability  jistributions^ 

Probabilistic  outcome  values  were  assigned  to  each  action 
outcome  to  provide  operational  realism,  and  also,  to  give  a  range  of 
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values  of  the  actions  selected.  These  values  then  could  be  used  to 
provide  part  of  the  basis  for  evaluation  of  the  commanders  performance 
in  selecting  appropriate  actions.  The  outcome  values  were  varied  as 
a  function  of  kind  of  weapons  chosen,  kind  of  armament,  number 
assigned,  time  assigned,  and  time  remaining  to  intercept.  These 
conditions  provided  a  large  distribution  of  action  alternatives  which 
presumably  was  searched  by  the  commander  before  he  made  his  action 
choices.  This  search  was  to  be  made  within  established  rule  structure 
provided  by  training  and  instructions. 

To  make  general  statements  about  commander's  performance  in 
search  of  the  action  alternative  domain,  it  is  necessary  that  the  prob¬ 
ability  outcome  for  each  action  be  the  same  for  all  subjects.  That  is 
to  say,  for  each  time  the  "ith"  weapon  combination  is  selected,  the 
subject  in  question  will  get  the  same  binary  outcome  (kill  or  non-kill). 
Thus,  each  outcome  will  influence  the  total  in  a  systematic  rather  than 
a  random  fashion.  Each  subject  will  get  the  same  result  each  time  he 
decides  to  use  a  particular  combination. 

The  use  of  "static"  outcome  distributions  also  expedites  the 
refereeing  process.  Since  an  average  of  around  100  action  combinations 
of  assignments  were  possible  for  each  track,  a  ready-access  program 
was  necessary  to  provide  the  "real-time"  servicing  of  the  commander's 
action  choices.  This  large  distribution  of  "possible"  actions  also  pre¬ 
cluded  any  significant  learning  by  the  commander,  e.  g.  ,  that  a  particu¬ 
lar  choice  in  the  sequence  will  always  kill  or  fail  to  kill. 

To  establish  this  distribution,  a  table  of  kill  probabilities  ranging 
from  .  2  to  .  9  (in  steps  of  .1)  was  generated  by  drawing  1000  samples  at 


each  probability  value  from  the  random  number  tables  of  the  Rand 
Corporation.  The  .1  probability  level  was  generated  by  reversing  the 
outcome  for  the  .9  value.  First,  the  observed  value  was  calculated 
for  each  100  unit  sample  for  each  probability  value,  and  then  the  grand 
mean  was  calculated.  The  range  and  mean  for  each  probability  value 
are  listed  below: 


■TABLE  ..1 

Range  and  Mean  for  Each  Probability  Value  Sample 


P  roba - 
bility 

0.  2 

0.  3 

0.  4 

0.  5 

0.  6 

0.  7 

0.8 

0.9 

0.  14 

0.28 

0.  3i 

0.45 

0.  54 

0.  65 

0.76 

0.  90 

Range 

0.  28 

SLASL 

0*41 

0.  53 

0.  69 

o*io. 

Q.97 

Mean 

0.  20 

0.  315 

0.  417 

0.488 

0.  596 

0.  703 

0.  809 

0.  933 

The  probability  outcome  for  each  action  then  was  applied  to  the 
referee  sheets  for  each  track  as  follows:  If  the  weapon  could  match 
the  hostile,  three  successive  samples  of  the  one,  two,  or  three-on -one 
weapons  per  salvo  were  permitted  when  assignment  was  early  enough 
to  achieve  maximum  kill  probability.  This  was  repeated  also  for  the 
second  interval  scramble  (later  than  five  minutes  of  track  life).  This 
sampling  was  repeated  across  armament  configuration  for  both  miss¬ 
iles  and  aircraft. 

(c)  Referee  scoring  procedures 

The  role  of  the  referee  action  (outcome-feedback 
processing)  will  be  apparent  if  two  representative  actions  are  followed 
through  processing  to  final  outcome.  Events  such  as  mis-matches 
are  obvious  --  the  weapons  are  scrambled,  show  on  the  board  as 
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as  committed;  then,  after  ten-minutes,  the  commander  is  informed 
that  the  action  failed  because  the  weapons  were  not  able  to  reach  their 
target.  Such  mis-matched  weapons  then  were  placed  on  airborne- 
available  status. 

It  may  be  assumed  the  commander  then  would  decide  to  re¬ 
assign  against  that  track.  His  action  order  might  be  "Track  73,  C-L.3, 
Priority  1.  "  This  order  designated  the  track  number,  directed  that 
three  missiles  be  committed  from  missile  site  C  in  the  long-range 
mode,  and  stated  that  this  was  a  first  priority  target  under  attack. 

The  scramble  recorder  logged  this  information  on  a  buck-slip,  along 
with  the  time  of  the  assignment,  which  he  read  from  a  digital  clock 
to  the  nearest  second.  He  passed  the  buck-slip  to  the  weapon-status 
clerk,  who  called  the  weapon-commitment  board  to  reduce  missile 
site  C  by  three  missiles.  Then  the  weapon-status  clerk  passed  the 
buck-slip  to  the  appropriate  aircraft  referee.  This  referee  entered 
the  missile  assignment  on  the  track  referee  sheet  for  future  refer¬ 
ence,  e.  g.  ,  in  case  he  received  a  subsequent  fighter  assignment  on 
this  track.  If  one  occurred,  he  checked  the  missile-assigned  board 
to  determine  outcome.  If  the  track  was  indicated  as  already  killed, 
he  returned  the  fighter  to  airborne -available  status.  If  the  outcome 
was  negative,  he  processed  the  outcome  in  normal  fashion. 

When  the  missile  referee  received  the  buck- slip  he  immediately 
posted  the  track  in  the  priority  one  column,  under  missile  C,  and 
reduced  the  total  of  weapons  at  that  site  by  three  to  indicate  that  the 
missiles  had  been  fired.  The  outcome  was  processed,  and  at  inter¬ 
cept  time  this  outcome  was  indicated  by  scrubbing  the  track  from  the 
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missile  assigned  board.  At  this  point  the  close-out  clerk  removed 
the  particular  track  from  the  system  digital  display  to  indicate  a  kill 
and  sent  a  chit  indicating  this  kill  to  the  commanders  technician  for 
relay  to  the  commander  upon  request. 

In  the  case  of  a  fighter  assignment,  the  sequence  would  be  as 
follows:  The  commander  might  report,  "Track  73,  Green  5,  IR2, 
Priority  1.  "  This  would  indicate  that  he  wanted  two  Green  fighters 
armed  with  IR  rockets  to  attack  from  airfield  five.  The  scramble 
recorder  logged  this  message  and  the  time  issued,  then  handed  the 
buck-slip  to  the  weapons-status  clerk.  This  clerk  called  the  weapon- 
commitment  board  and  reduced  the  IR-rocket  capability  at  Green  five 
by  two  and  also  indicated  on  the  buck- slip  that  one  aircraft  was  pre¬ 
programmed  to  be  lost  as  a  result  of  the  engagement.  Lost  fighter 
probabilities  due  to  engagement  were  .  5  for  Red,  .  3  for  Blue,  and  .  2 
for  Green.  He  then  passed  the  slip  to  the  appropriate  aircraft  referee. 

Aircraft  referee  number  two  had  responsibility  for  the  upper 
half  of  the  track  numbers,  so  he  processed  the  assignment.  He  re¬ 
corded  the  time  on  the  referee  sheet  for  later  scoring  purposes, 
added  the  intercept  time  (11  minutes)  to  the  assigned  time,  indicated 
that  the  outcome  was  a  kill,  and  added  twice  the  intercept  time  to  the 
assign  time.  This  last  step  served  to  indicate  that  this  weapon  should 
go  into  the  turn  around  position  on  the  status  board  to  simulate  the 
time  delay  for  rearming  and  refueling.  A  small  chit  then  was  passed 
to  the  close-out  clerk  who  was  responsible  for  sequencing  these  slips 
for  track  close-out  at  the  proper  time.  Also  he  would  send  a  notice 
to  the  commanders  technician  to  indicate  a  kill.  The  initial  buck-slip 
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was  returned  to  a  weapons  buffer  storage  clerk  who  held  it  until  the 
time  indicated  to  tote  the  lost  fighter.  Then,  the  buck-slip  was  passed 
to  the  weapon- status  clerk,  and  he  posted  one  lost  IR  weapon  at  the 
Green  five  airfield  location.  The  clerk  who  arranged  the  slips  in  time 
again  held  the  buck- slip  until  the  turn  around  time  indicated.  Weapon 
status  was  notified  at  that  time  and  posted  a  Green  five  IR  weapon  in 
the  turn  around  position.  The  commander  was  aware  that  after 
fifteen  minutes  in  this  location  the  weapon  would  return  to  ground- 
available  for  further  usage. 

C.  Experimental  Design 

The  first  major  experiment  was  planned  to  consider  action 
selection  performance  in  a  situation  providing  reliable  surveillance 
data  and  reliable  outcome  processing.  Of  course,  decision  perform¬ 
ance  can  be  expected  to  be  degraded  by  any  unreliability  that  may 
enter  the  information  processing  sequence.  But,  before  initiating 
experimental  work  with  false  tracks,  incorrect  identity  information, 
or  any  of  the  other  "noise"  factors  that  may  be  expected  to  creep  into 
any  real  surveillance  system,  the  first  experiment  was  directed 
toward  the  task  of  famishing  "baseline"  information  decision  success. 
In  a  way  this  might  be  thought  of  as  a  search  for  normal  decision- 
performance  before  bringing  in  "gremlin"  factors  that  past  experience 
had  shown  may  crop -up  in  battle. 

Since  previous  studies  had  not  furnished  a  secure  base  from 
which  to  predict  the  effect  of  load  stresses  upon  performance  in  a 
decision  situation  approaching  the  present  one  in  level  of  complexity, 
the  experimental  design  was  prepared  to  cover  a  wide  span  of  task 


-137- 


loads.  To  meet  this  objective  it  was  felt  desirable  to  investigate  four 
levels  of  track  load,  two  levels  of  threat  complexity,  two  weapon  siting 
configurations  and  to  a  lesser  extent,  two  groups  of  subjects.  The  difference 
in  the  subjects  was  that  one  group  had  received  fewer  experimental  runs  in 
serving  under  only  one  level  of  weapon- site  configuration.  Hence,  they 
may  be  considered  less  experienced. 

The  design  presented  below  was  intended  to  permit  both  the  subjects 

and  the  referees  to  function  in  a  reasonable  manner  and  to  permit  comple- 

I 

tion  of  the  data  collection  trials  in  minimal  time.  Time  was  a  concern  due. 
to  the  difficulties  of  scheduling  and  retaining  military  subjects  who  had 
other  duties  to  perform,  because  of  cost  factors  and  other  problems  assoc¬ 
iated  with  the  real-time  simulation  of  complex  problems.  This  design 
called  for  each  subject  to  progress  through  various  combinations  of  condi¬ 
tions  under  investigation  in  a  stepwise  fashion;  as  his  experience  increased 
in  the  experiment,  he  faced  heavier  task  loads. 

This  plan  was  not  a  treatment  sequence  in  terms  of  classical  statisti¬ 
cal  design  (22).  To  investigate  fully  all  combinations  of  experimental 
wiriables  in  random  order  would  have  been  prohibitive,  both  in  terms  of 
time  and  cost.  The  interest  of  this  investigation  was  in  the  practical  or 
indicative  types  of  results,  rather  than  the  inferential,  statistical  type. 

To  a  classicist  who  would  ask  when  is  one  justified  to  do  studies  that  are 
less  than  complete  experiments,  the  reply  might  be  in  order  -- 
"Quantification  and  methodological  sophistication  are  late  products  of  any 
science  and  as  such  they  should  be  long  range  goals:  mistaking  them  for 
proximal  goals  can  render  a  science  impotent".  (23) 
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♦First  digit  is  the  tape  number,  the  second  digit  indicates  number  of  sites. 


APPENDIX  11. 
TABLE  3 

Experimental  Design 


Sites 

Group  I 


N=6  5 

Mission 

Threat 

Level  I  8 

Threat  5 

Level  II  8 


Group  II 
N=3 
Threat 

Level  I  8 

Threat 

Level  II  8 


♦The  run  sequence  is  to  read 
vertically  within  each  subject 
group. 
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